On Mon, Jul 19, 2021 at 9:43 AM Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:
> It looks like it is important to have shrx for x86 which appears only when -march=x86-64-v3 is used (see
https://github.com/golang/go/issues/47120#issuecomment-877629712 ).
> Just in case: I know x86 wound not use fallback implementation, however, the sole purpose of shift-based DFA is to fold all the data-dependent ops into a single instruction.
I saw mention of that instruction, but didn't understand how important it was, thanks.
> An alternative idea: should we optimize for validation of **valid** inputs rather than optimizing the worst case?
> In other words, what if the implementation processes all characters always and uses a slower method in case of validation failure?
> I would guess it is more important to be faster with accepting valid input rather than "faster to reject invalid input".
> static int pg_utf8_verifystr2(const unsigned char *s, int len) {
> if (pg_is_valid_utf8(s, s+len)) { // fast path: if string is valid, then just accept it
> return s + len;
> }
> // slow path: the string is not valid, perform a slower analysis
> return s + ....;
> }
That might be workable. We have to be careful because in COPY FROM, validation is performed on 64kB chunks, and the boundary could fall in the middle of a multibyte sequence. In the SSE version, there is this comment:
+ /*
+ * NB: This check must be strictly greater-than, otherwise an invalid byte
+ * at the end might not get detected.
+ */
+ while (len > sizeof(__m128i))
...which should have more detail on this.
--
John Naylor
EDB:
http://www.enterprisedb.com
