CentreForDigitalHumanities / CentreForDigitalHumanities/tscan
Frog output isn't received: Empty result for FoLiaParsing
- Dominant language
- C++
- Stars
- 19
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
With some complicated texts (probably) the Frog output isn't received by T-Scan from Frog. Resulting in an empty document.
It starts here:
https://github.com/UUDigitalHumanitieslab/tscan/blob/1ae5af3f22929366b3d58a66dddb324153e76d2c/src/tscan.cxx#L3126-L3130
A connection is made with Frog, using this connection the plain text is send to Frog.
The text is then closed using `EOT` and Frog starts processing it. The response should then be read and written to `result`.
```cxx
client.write( "\nEOT\n" );
string result;
string s;
while ( client.read( s ) ) {
if ( s == "READY" )
break;
result += s + "\n";
}
```
> https://github.com/UUDigitalHumanitieslab/tscan/blob/1ae5af3f22929366b3d58a66dddb324153e76d2c/src/tscan.cxx#L3194C6-L3201
This will contain the FoliaXML in plain text which can then be loaded using
```cxx
doc = new folia::Document();
try {
doc->readFromString( result );
#ifdef DEBUG_FROG
cerr << "finished" << endl;
#endif
}
```
> https://github.com/UUDigitalHumanitieslab/tscan/blob/1ae5af3f22929366b3d58a66dddb324153e76d2c/src/tscan.cxx#L3210C1-L3216
The problem is however that the `result` is empty! Now this could be a problem with Frog.
So let's use Telnet:
```bash
$ telnet localhost 7001
Dit is een hele lang zin die niet gepakt wordt, dit is niet de zin, vraag mij om de zin en dan kan ik hem doorsturen.
EOT
```
And you get the response as expected:
```
READY
```
It may take a little while so it could perhaps be some race/timing issue.
I've tried giving it some more time e.g. adding a sleep before reading. I've also tried reading again after the initial attempt was empty after a sleep. Here's an attempt of me complicating the line ` while ( client.read( s ) ) {` into this:
```cxx
client.setNonBlocking();
// wait at least a little bit for the response
usleep( 100 );
while ( true ) {
if ( !client.read( s ) ) {
if ( !result.empty() ) {
// done reading
break;
}
else {
cerr << "giving Frog some more time... ";
sleep( 30 );
if ( !client.read( s ) ) {
// done reading
cerr << "still nothing" << endl;
break;
}
else {
cerr << s.size() << endl;
}
}
}
```
Anyway, it doesn't work or I wouldn't be writing this issue down. I'm going on leave for a month but I'm not quite sure how to fix this. _Possibly_ it's some bug in the socket implementation, maybe a work-around could be to simply use a different socket library. A super low tech (slow) approach could also be to call Frog from the command line which should also work, but is really slow because Frog will have to start up for every document. Maybe this could be moved to the Python service and let that deal with this. (Prepare all the input as FoliaXML files).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/tscan.cxx at the Frog connection around lines 3126-3130 and the response loop around lines 3194-3201; inspect how client.read() handles the response and READY marker. Reproduce the empty result with a complicated text and compare it with the Telnet response. Done means Frog output is reliably received and the FoliaXML is parsed into a document.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100