The task was to razdobit about 60GB string data. Unique among them, about 25-30%
Decided to use a mysql unique index for this.
1. Unique is better to make a field with the row (1-5 words) or better take first the crc32 of this string, and the hash is to hang a unique index?
2. Is it possible to use consternee semblance of particioniranja, but not at the table level and at the DB level?
For example, to divide the data by the first letter of the string (get 28 physical bases), and at the same time to only complete one of them, thereby reducing the consumption of RAM?