fix: comprehensive audit, bug fixes, and documentation overhaul

[JarbasAl]

Summary

• Bug Fixes:
  • Fixed paragraph_tokenize crash when input starts with a newline.
  • Fixed get_empty_spans indexing and content for trailing spaces.
  • Added regex escaping for delimiters in the chunk function.
• Improvements:
  • Overhauled _WORD_REGEX to robustly handle version numbers, emails, and hyphenated words while maintaining compatibility with negative number tests.
  • Refactored span_indexed_sentence_tokenize and span_indexed_paragraph_tokenize to use more robust string-finding logic.
• Documentation:
  • Created a new /docs directory with GETTING_STARTED.md and API.md.
  • Modernized the main README.md.
• Testing:
  • Added an extensive test suite in test/test_extensive.py covering numerous edge cases.

Test plan

• Run existing tests: PYTHONPATH=. pytest test/test_chunking.py
• Run new extensive tests: PYTHONPATH=. pytest test/test_extensive.py
• Verify documentation links and content.

AI Usage Disclaimer

Gemini 2.0 Flash. Human reviewed before merge.

🤖 Generated with Gemini CLI

[coderabbitai]

Warning

Review limit reached

@JarbasAl, we couldn't start this review because you've used your available PR reviews for now.

Your plan currently allows 1 review/hour. Refill in 50 minutes and 36 seconds.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more review capacity refills, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than trial, open-source, and free plans. In all cases, review capacity refills continuously over time.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0fb93bfa-2273-4187-a372-efda70ce437c

📥 Commits

Reviewing files that changed from the base of the PR and between a4bd72f and 7589cf0.

📒 Files selected for processing (6)

• docs/API.md
• docs/GETTING_STARTED.md
• quebra_frases/chunks.py
• quebra_frases/tokenization.py
• readme.md
• test/test_extensive.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/audit-and-docs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}