Splitting text into sentences is not a trivial - this is why you should add an extra space after the sentences

Here are some existing tools for splitting text into sentences:


None of them is recognizing the end of sentences 100% correctly.

Here is an example (from this blog):

Now you should know why you should enter two spaces after the sentences. Some programs rely on this.

In the computer era, spacing between sentences is handled in several different ways by various software packages. Some systems accept whatever the user types, while others attempt to alter the spacing, or use the user input as a method of detecting sentences. Computer-based word processors, and software such as TeX allow users to arrange text in a manner previously only available to professional typesetters.[71]

The text editing environment in Emacs uses a double space following a period to identify the end of sentences unambiguously; the double space convention prevents confusion with periods within sentences that signify abbreviations. How Emacs recognizes the end of a sentence is controlled by the settings sentence-end-double-space and sentence-end.[72] The vi editor also follows this convention; thus, it is relatively easy to manipulate (jump over, copy, delete) whole sentences in both emacs and vi.

Wikipedia: Sentence spacing - Digital age

When you type two spaces in a HTML code the extra space is not displayed.

Recommended articles:
Testing out the NLTK sentence tokenizer
How to Split Sentences

Comments

Post a Comment

Popular posts from this blog

Do not use (only) flash memory (SSD drives, hardware wallets, USB flash drives) for your precious private keys!

Image
Flash memory is not a reliable medium for archives. Especially when there is no regular power. I have personal experience with usb flash drive not powered for weeks - one file became corrupted . (Read more here: Archiving private keys - TLDR version. ) You should always back up on paper and other mediums. Flash memory is prone to failure if it is not powered for weeks or more and if there are ionizing radiation When you write your precious private keys you should use technologies like Parchive and ZFS . And make several copies of your files. It's OK if you use your USB flash drive for another backup, but don't rely on it! Always back up on DVDs (even small files!), paper and online (after encryption with CPU and RAM intensive key derivative function like scrypt). Here is example of using the scrypt utility: $ sudo apt-get install scrypt $ scrypt enc -M 1073741824 -t 200 secret.txt encrypted.scrypt Do not use default values of "-M" and "-t", they...
Read more

Archiving private keys - TLDR version

0. Make multiple encrypted copies. On DVDs (they are better than CDs and Blu-Ray discs; DVD+R are better than DVD-R), paper, cloud services like DropBox, OneDrive, Google Drive, e-mail it to yourself and to your friends, use P2P storage services like MaidSafe, Storj and Sia , etc. 1. Use proper font when printing PGP encrypted keys on paper. 2. Flash memory (SSD, USB flash drives, hardware wallets) is less reliable when not powered regularly (i.e. every week). 3. Use error correction methods like Parchive and ZFS. 4. Print on paper or store on digital media only encrypted data. 5. Your encryption software should use CPU/RAM-intensive KDF (i.e. scrypt with secure options - do not use defaults! ). First, encrypt with scrypt and then encrypt it again with PGP (using different password!) in ASCII armor mode before print it (other methods like QR codes may not be reliable as multiple copies of the PGP ASCII armor). Do not use the same password for the PGP because it's easy to brute...
Read more
[ad removed]