[Bug] The tokenizer will recognize the "Birds" as a <unk> and an "s"

[Zxl19990529]

Related issues: #92 #66

This issue happens in the collate_fn function of the file utils/dataset.py during the validation phase. I finally found this issue roots in the bug of the tokenizer when run into the image of "7939894288_3028c8874a_o.jpg".
The original corresponding text of this image is:

The birds have various ways of searching for food. What part of their body helps them to grab and pick up food from the ground in the picture?

After adding the prompt before line 94, the variable "conversation" in line 95 is:

A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <im_start><im_end>
Birds have various ways of searching for food. What part of their body helps them to grab and pick up food from the ground in the picture? Please output segmentation mask. ASSISTANT: [SEG].

Meanwhile, the tokenizer will encoder that text into the following ids:

tensor([    1,   319, 13563,  1546,   263, 12758,  5199,   322,   385, 23116,
        21082, 20255, 29889,   450, 20255,  4076,  8444, 29892, 13173, 29892,
          322,  1248,   568,  6089,   304,   278,  5199, 29915, 29879,  5155,
        29889,  3148,  1001, 29901, 32001,  -200, 32002,     0, 29879,   505,
         5164,  5837,   310, 11975,   363,  9687, 29889,  1724,   760,   310,
         1009,  3573,  6911,   963,   304, 17229,   322,  5839,   701,  9687,
          515,   278,  5962,   297,   278,  7623, 29973,  3529,  1962, 10768,
          362, 11105, 29889,   319,  1799,  9047, 13566, 29901, 32000, 29889,
            2])

Note that, the 32001, -200, 32002 are the <im_start><image><im_end>, after that, there is a zero, see that? That's the problem here!
Now, let's decode back those ids using the tokenizer.decode() function:

print(tokenizer.decode([32002,     0, 29879,   505,
         5164,  5837,   310, 11975,   363,  9687, 29889,  1724,   760,   310,
         1009,  3573,  6911,   963,   304, 17229,   322,  5839,   701,  9687,
          515,   278,  5962,   297,   278,  7623, 29973,  3529,  1962, 10768,
          362, 11105, 29889,   319,  1799,  9047, 13566, 29901, 32000, 29889,
            2]))

then we will get:

<im_end> <unk>s have various ways of searching for food. What part of their body helps them to grab and pick up food from the ground in the picture? Please output segmentation mask. ASSISTANT: [SEG]  .</s>

So, the word "Birds" are split into a "" and an "s", where the ids are 0 and 29879. However, the 0 is a special token for padding, this will result in the total_len variable in line 96 total_len = int(target.ne(tokenizer.pad_token_id).sum()) is smaller than the total token length, and finally lead to an error in line 135: assert cur_len == total_len

[Zxl19990529]

In my project, I solve this problem by modifying the original text in 7939894288_3028c8874a_o.json with an additional "The" before "Brids":

"The birds have various ways of searching for food. What part of their body helps them to grab and pick up food from the ground in the picture?"

[Zxl19990529]

The blowing samples have the same problem:

84 != 83, for image ./dataset/reason_seg/ReasonSeg/val/scene0684_00_0.jpg
tensor([[    1,   319, 13563,  1546,   263, 12758,  5199,   322,   385, 23116,
         21082, 20255, 29889,   450, 20255,  4076,  8444, 29892, 13173, 29892,
           322,  1248,   568,  6089,   304,   278,  5199, 29915, 29879,  5155,
         29889,  3148,  1001, 29901, 32001,  -200, 32002,     0,   278,  4337,
         29892,   825,  1203,   508,   367,  1304,   304,  3787,   322,  8608,
          5164,   480,   299,  2722,   322,  2319, 22329,  4452, 29892,   607,
           338,   380,   332,  4518,   322, 13774,  4780,   304,  8677, 29973,
          3529,  1962, 10768,   362, 11105, 29889,   319,  1799,  9047, 13566,
         29901, 32000, 29889,     2]])
91 != 90, for image ./dataset/reason_seg/ReasonSeg/val/5247939212_ca5c8a4379_o.jpg
tensor([[    1,   319, 13563,  1546,   263, 12758,  5199,   322,   385, 23116,
         21082, 20255, 29889,   450, 20255,  4076,  8444, 29892, 13173, 29892,
           322,  1248,   568,  6089,   304,   278,  5199, 29915, 29879,  5155,
         29889,  3148,  1001, 29901, 32001,  -200, 32002,     0, 29892,   746,
           263,  1559,   338, 14089,   287,   373,   278,  2625,   310,   278,
          6520, 29892,   278,  7156,  1122, 11423,   635,  5967,   278,  1559,
          3050,  1722, 29889,  1724,   760,   297,   278,  7623,   508, 12266,
           393,   278,  1559,  3050,   338,  1722, 29973,  3529,  1962, 10768,
           362, 11105, 29889,   319,  1799,  9047, 13566, 29901, 32000, 29889,
             2]])
84 != 83, for image ./dataset/reason_seg/ReasonSeg/val/8087061900_51c823704a_o.jpg
tensor([[    1,   319, 13563,  1546,   263, 12758,  5199,   322,   385, 23116,
         21082, 20255, 29889,   450, 20255,  4076,  8444, 29892, 13173, 29892,
           322,  1248,   568,  6089,   304,   278,  5199, 29915, 29879,  5155,
         29889,  3148,  1001, 29901, 32001,  -200, 32002,     0, 29879,  4049,
           817,   263,  2058,   304,  1791,   470, 14111,  1009,  8388,   618,
           886, 29889,  1724,   760,   310,   263,  5447,   297,   278,  7623,
         16688,   263, 13907,  9758,   363, 17952,   304,   437,   577, 29973,
          3529,  1962, 10768,   362, 11105, 29889,   319,  1799,  9047, 13566,
         29901, 32000, 29889,     2]])
81 != 80, for image ./dataset/reason_seg/ReasonSeg/val/22758113_b762c61e62_o.jpg
tensor([[    1,   319, 13563,  1546,   263, 12758,  5199,   322,   385, 23116,
         21082, 20255, 29889,   450, 20255,  4076,  8444, 29892, 13173, 29892,
           322,  1248,   568,  6089,   304,   278,  5199, 29915, 29879,  5155,
         29889,  3148,  1001, 29901, 32001,  -200, 32002,     0,  4285,   526,
          4049,  1592, 16242,   411,  5164,  8492,   322, 25340,   304,  8677,
           714,  1009, 22204, 29889,  1724,  2944,   297,   278,  7623,   508,
           367,   770,  2164,   408,   263, 28639, 29973,  3529,  1962, 10768,
           362, 11105, 29889,   319,  1799,  9047, 13566, 29901, 32000, 29889,
             2]])
78 != 77, for image ./dataset/reason_seg/ReasonSeg/val/207090875_9a57789fae_o.jpg
tensor([[    1,   319, 13563,  1546,   263, 12758,  5199,   322,   385, 23116,
         21082, 20255, 29889,   450, 20255,  4076,  8444, 29892, 13173, 29892,
           322,  1248,   568,  6089,   304,   278,  5199, 29915, 29879,  5155,
         29889,  3148,  1001, 29901, 32001,  -200, 32002,     0,   292,   338,
           263,  5972,  6354,   363, 26681,   362,   322,   454,   275,   545,
         29889,  1724,  5780,   338,   278,   767,   297,   278,  7623,   773,
           304,  4380,  9427, 29973,  3529,  1962, 10768,   362, 11105, 29889,
           319,  1799,  9047, 13566, 29901, 32000, 29889,     2]])
88 != 87, for image ./dataset/reason_seg/ReasonSeg/val/764271537_cf309cdeaf_o.jpg
tensor([[    1,   319, 13563,  1546,   263, 12758,  5199,   322,   385, 23116,
         21082, 20255, 29889,   450, 20255,  4076,  8444, 29892, 13173, 29892,
           322,  1248,   568,  6089,   304,   278,  5199, 29915, 29879,  5155,
         29889,  3148,  1001, 29901, 32001,  -200, 32002,     0, 29892,  4094,
           454, 10327,   515, 12570, 29891,   715,  3774,   292,   470,   285,
         14766,  1691, 29889,  1724,   760,   310,   278,   715,  3774,   292,
          1788,   297,   278,  7623,   508,   367,   263,  7037,  2752,   310,
           278,  4094, 24993, 29973,  3529,  1962, 10768,   362, 11105, 29889,
           319,  1799,  9047, 13566, 29901, 32000, 29889,     2]])
76 != 75, for image ./dataset/reason_seg/ReasonSeg/val/889861234_40005de024_o.jpg
tensor([[    1,   319, 13563,  1546,   263, 12758,  5199,   322,   385, 23116,
         21082, 20255, 29889,   450, 20255,  4076,  8444, 29892, 13173, 29892,
           322,  1248,   568,  6089,   304,   278,  5199, 29915, 29879,  5155,
         29889,  3148,  1001, 29901, 32001,  -200, 32002,     0,  4049,   788,
         10489,   304,  4094,   304, 11157,   967, 21779, 29889,  1724,   760,
           310,   278,  7623,   674,   367, 11233,   287,   408,   263,   367,
         19698, 29973,  3529,  1962, 10768,   362, 11105, 29889,   319,  1799,
          9047, 13566, 29901, 32000, 29889,     2]])
96 != 95, for image ./dataset/reason_seg/ReasonSeg/val/20026538219_9fbd8d2024_o.jpg
tensor([[    1,   319, 13563,  1546,   263, 12758,  5199,   322,   385, 23116,
         21082, 20255, 29889,   450, 20255,  4076,  8444, 29892, 13173, 29892,
           322,  1248,   568,  6089,   304,   278,  5199, 29915, 29879,  5155,
         29889,  3148,  1001, 29901, 32001,  -200, 32002,     0,  1181,   397,
           270, 17006,  4049,  3160,   263, 18806, 29891,  2148,  2073,   393,
         26371,  2925,   278, 21054,   272, 29889,  1724,  2944,   297,   278,
          7623,   508,   367,   269,   802,  6096,   287, 11480,   278,  7205,
          1181,   397,   408,   263, 18806, 29891, 21054,   272,   427,  5403,
          2265, 29973,  3529,  1962, 10768,   362, 11105, 29889,   319,  1799,
          9047, 13566, 29901, 32000, 29889,     2]])
84 != 83, for image ./dataset/reason_seg/ReasonSeg/val/400894499_50d82fd433_o.jpg
tensor([[    1,   319, 13563,  1546,   263, 12758,  5199,   322,   385, 23116,
         21082, 20255, 29889,   450, 20255,  4076,  8444, 29892, 13173, 29892,
           322,  1248,   568,  6089,   304,   278,  5199, 29915, 29879,  5155,
         29889,  3148,  1001, 29901, 32001,  -200, 32002,     0,   263,   592,
           284, 29892,  2305, 12234,   671,  3477,   575,  2719,   304,  6963,
          9687,   304,  1009, 13394, 29879, 29889,  1724,  5780,   297,   278,
          7623,   508,   367,  1304,   304,  6095,  5589,   445,   817, 29973,
          3529,  1962, 10768,   362, 11105, 29889,   319,  1799,  9047, 13566,
         29901, 32000, 29889,     2]])
81 != 80, for image ./dataset/reason_seg/ReasonSeg/val/33343263_c4f465d900_o.jpg
tensor([[    1,   319, 13563,  1546,   263, 12758,  5199,   322,   385, 23116,
         21082, 20255, 29889,   450, 20255,  4076,  8444, 29892, 13173, 29892,
           322,  1248,   568,  6089,   304,   278,  5199, 29915, 29879,  5155,
         29889,  3148,  1001, 29901, 32001,  -200, 32002,     0,  3064,   310,
          1370, 29892,  5075,  4395, 24413,   526, 15574,  1304,   304, 12566,
         13936,   322,  3033,   482,   297, 15499, 29889,  1724,  1203,   297,
           278,  7623,   508,  3867,  1316, 13047, 29973,  3529,  1962, 10768,
           362, 11105, 29889,   319,  1799,  9047, 13566, 29901, 32000, 29889,
             2]])