Take a look at a fingerspelling chart:
\nhttps://www.startasl.com/fingerspelling/
\nspecifically, look at the letters A T N M S in order. Note how the thumb starts outside the fist with A and then progresses to being under the first (T), second (N), and third finger(M). With S it is in front of the hand.
\nNow go to https://asl.ms/()/spellit_large.htm and enter
\natnmsatnmsatnmsatms
\nTurn the speed up. How well can you see the subtle differences? How well can hand posture trackers detect these subtle differences?
\nExpert signers use context to help recognize which letter is being signed as the movement may be too fast, the lighting might be too poor, or the fingerspeller's dexterity might be limited. We can do something similar by looking at transition between letters to give clues as to what was spelled.
\nIn my early research in cursive handwriting recognition using hidden Markov models I used \"contexts\" (recognizing groups of two, three, or even four letters together as one model instead of individual letters).
\nOn-line cursive handwriting recognition using speech recognition methods
\nIt helps with co-articulation, where the previous and the next letter affects the appearance of the current letter. It also helps where a letter itself might be hard to identify, but the letters around it make it clear what is meant (for those who are familiar with cursive, compare the wi in wick versus the ui in quick). Prior probabilities can help weight recognition towards more common groups of letters, which also automatically happens to some extent based on the amount of representation groups of letters have in a training set.
\nStochastic grammars further help to reduce the \"perplexity\" of the problem, as certain words are more likely to follow other words. For example, the \"quick brown fox\" is more likely than \"fox brown quick.\" There are large public text datasets that can aid in creating such grammars.
\nHere is another example that shows the value of contexts and grammar: NT is a more common combination of letters than TN when in the middle of a word. Off-hand, I can not even think of a word with TN in the middle. However, for addresses, TN may appear often, as it is the abbreviation for a state. Seeing \"Memphis, \" may predict an upcoming \"TN\" where \"MOVEME\" might foreshadow an upcoming \"NT\"
\nOf course, given a big enough neural net model and enough training data, these details could be learned. However, for smaller models and smaller data sets, finding a way to apply contextual knowledge may be a more efficient way to improve accuracy. Perhaps pre-training a neural net on text datasets might help? Or perhaps finding a way to apply contexts and stochastic grammars may be better? Offhand, I do not know of any papers that compare these methods and whether, in the limit, they are equivalent. However, I bet adding context in some way can be powerful in helping resolve when a signer is talking about ANTMAN or SANTAS.
", "messages": [ { "id": 2300297, "postDate": "2023-06-13T05:58:05.357Z", "content": "Take a look at a fingerspelling chart:
\nhttps://www.startasl.com/fingerspelling/
\nspecifically, look at the letters A T N M S in order. Note how the thumb starts outside the fist with A and then progresses to being under the first (T), second (N), and third finger(M). With S it is in front of the hand.
\nNow go to https://asl.ms/()/spellit_large.htm and enter
\natnmsatnmsatnmsatms
\nTurn the speed up. How well can you see the subtle differences? How well can hand posture trackers detect these subtle differences?
\nExpert signers use context to help recognize which letter is being signed as the movement may be too fast, the lighting might be too poor, or the fingerspeller's dexterity might be limited. We can do something similar by looking at transition between letters to give clues as to what was spelled.
\nIn my early research in cursive handwriting recognition using hidden Markov models I used \"contexts\" (recognizing groups of two, three, or even four letters together as one model instead of individual letters).
\nOn-line cursive handwriting recognition using speech recognition methods
\nIt helps with co-articulation, where the previous and the next letter affects the appearance of the current letter. It also helps where a letter itself might be hard to identify, but the letters around it make it clear what is meant (for those who are familiar with cursive, compare the wi in wick versus the ui in quick). Prior probabilities can help weight recognition towards more common groups of letters, which also automatically happens to some extent based on the amount of representation groups of letters have in a training set.
\nStochastic grammars further help to reduce the \"perplexity\" of the problem, as certain words are more likely to follow other words. For example, the \"quick brown fox\" is more likely than \"fox brown quick.\" There are large public text datasets that can aid in creating such grammars.
\nHere is another example that shows the value of contexts and grammar: NT is a more common combination of letters than TN when in the middle of a word. Off-hand, I can not even think of a word with TN in the middle. However, for addresses, TN may appear often, as it is the abbreviation for a state. Seeing \"Memphis, \" may predict an upcoming \"TN\" where \"MOVEME\" might foreshadow an upcoming \"NT\"
\nOf course, given a big enough neural net model and enough training data, these details could be learned. However, for smaller models and smaller data sets, finding a way to apply contextual knowledge may be a more efficient way to improve accuracy. Perhaps pre-training a neural net on text datasets might help? Or perhaps finding a way to apply contexts and stochastic grammars may be better? Offhand, I do not know of any papers that compare these methods and whether, in the limit, they are equivalent. However, I bet adding context in some way can be powerful in helping resolve when a signer is talking about ANTMAN or SANTAS.
", "rawMarkdown": "Take a look at a fingerspelling chart:\n\n[https://www.startasl.com/fingerspelling/](https://www.startasl.com/fingerspelling/)\n\nspecifically, look at the letters A T N M S in order. Note how the thumb starts outside the fist with A and then progresses to being under the first (T), second (N), and third finger(M). With S it is in front of the hand.\n\nNow go to [https://asl.ms/()/spellit_large.htm](https://asl.ms/()/spellit_large.htm) and enter \n\natnmsatnmsatnmsatms\n\nTurn the speed up. How well can you see the subtle differences? How well can hand posture trackers detect these subtle differences?\n\nExpert signers use context to help recognize which letter is being signed as the movement may be too fast, the lighting might be too poor, or the fingerspeller's dexterity might be limited. We can do something similar by looking at transition between letters to give clues as to what was spelled.\n\nIn my early research in cursive handwriting recognition using hidden Markov models I used \"contexts\" (recognizing groups of two, three, or even four letters together as one model instead of individual letters). \n\n[On-line cursive handwriting recognition using speech recognition methods](https://ieeexplore.ieee.org/iel2/3104/8834/00389432.pdf?casa_token=EdbBFGdcXHAAAAAA:EAh_OnxFt-bB1GTrwy5GjSgCcZc-1zMsipbP0WqhAn6gbODtsh5Bmr_5gQWGvKymGz2J7qYLYw)\n\nIt helps with co-articulation, where the previous and the next letter affects the appearance of the current letter. It also helps where a letter itself might be hard to identify, but the letters around it make it clear what is meant (for those who are familiar with cursive, compare the wi in wick versus the ui in quick). Prior probabilities can help weight recognition towards more common groups of letters, which also automatically happens to some extent based on the amount of representation groups of letters have in a training set.\n\nStochastic grammars further help to reduce the \"perplexity\" of the problem, as certain words are more likely to follow other words. For example, the \"quick brown fox\" is more likely than \"fox brown quick.\" There are large public text datasets that can aid in creating such grammars.\n\nHere is another example that shows the value of contexts and grammar: NT is a more common combination of letters than TN when in the middle of a word. Off-hand, I can not even think of a word with TN in the middle. However, for addresses, TN may appear often, as it is the abbreviation for a state. Seeing \"Memphis, \" may predict an upcoming \"TN\" where \"MOVEME\" might foreshadow an upcoming \"NT\"\n\nOf course, given a big enough neural net model and enough training data, these details could be learned. However, for smaller models and smaller data sets, finding a way to apply contextual knowledge may be a more efficient way to improve accuracy. Perhaps pre-training a neural net on text datasets might help? Or perhaps finding a way to apply contexts and stochastic grammars may be better? Offhand, I do not know of any papers that compare these methods and whether, in the limit, they are equivalent. However, I bet adding context in some way can be powerful in helping resolve when a signer is talking about ANTMAN or SANTAS.\n\n", "votes": 13 } ], "comments": [], "raw_markdown_by_id": { "2300297": "Take a look at a fingerspelling chart:\n\n[https://www.startasl.com/fingerspelling/](https://www.startasl.com/fingerspelling/)\n\nspecifically, look at the letters A T N M S in order. Note how the thumb starts outside the fist with A and then progresses to being under the first (T), second (N), and third finger(M). With S it is in front of the hand.\n\nNow go to [https://asl.ms/()/spellit_large.htm](https://asl.ms/()/spellit_large.htm) and enter \n\natnmsatnmsatnmsatms\n\nTurn the speed up. How well can you see the subtle differences? How well can hand posture trackers detect these subtle differences?\n\nExpert signers use context to help recognize which letter is being signed as the movement may be too fast, the lighting might be too poor, or the fingerspeller's dexterity might be limited. We can do something similar by looking at transition between letters to give clues as to what was spelled.\n\nIn my early research in cursive handwriting recognition using hidden Markov models I used \"contexts\" (recognizing groups of two, three, or even four letters together as one model instead of individual letters). \n\n[On-line cursive handwriting recognition using speech recognition methods](https://ieeexplore.ieee.org/iel2/3104/8834/00389432.pdf?casa_token=EdbBFGdcXHAAAAAA:EAh_OnxFt-bB1GTrwy5GjSgCcZc-1zMsipbP0WqhAn6gbODtsh5Bmr_5gQWGvKymGz2J7qYLYw)\n\nIt helps with co-articulation, where the previous and the next letter affects the appearance of the current letter. It also helps where a letter itself might be hard to identify, but the letters around it make it clear what is meant (for those who are familiar with cursive, compare the wi in wick versus the ui in quick). Prior probabilities can help weight recognition towards more common groups of letters, which also automatically happens to some extent based on the amount of representation groups of letters have in a training set.\n\nStochastic grammars further help to reduce the \"perplexity\" of the problem, as certain words are more likely to follow other words. For example, the \"quick brown fox\" is more likely than \"fox brown quick.\" There are large public text datasets that can aid in creating such grammars.\n\nHere is another example that shows the value of contexts and grammar: NT is a more common combination of letters than TN when in the middle of a word. Off-hand, I can not even think of a word with TN in the middle. However, for addresses, TN may appear often, as it is the abbreviation for a state. Seeing \"Memphis, \" may predict an upcoming \"TN\" where \"MOVEME\" might foreshadow an upcoming \"NT\"\n\nOf course, given a big enough neural net model and enough training data, these details could be learned. However, for smaller models and smaller data sets, finding a way to apply contextual knowledge may be a more efficient way to improve accuracy. Perhaps pre-training a neural net on text datasets might help? Or perhaps finding a way to apply contexts and stochastic grammars may be better? Offhand, I do not know of any papers that compare these methods and whether, in the limit, they are equivalent. However, I bet adding context in some way can be powerful in helping resolve when a signer is talking about ANTMAN or SANTAS.\n\n" } }