CentreForDigitalHumanities / CentreForDigitalHumanities/tscan

Frog output isn't received: Empty result for FoLiaParsing

Open
#85 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
19
Forks
7
PR merge metrics
No merged PRs in 30d

Description

With some complicated texts (probably) the Frog output isn't received by T-Scan from Frog. Resulting in an empty document.

It starts here:

https://github.com/UUDigitalHumanitieslab/tscan/blob/1ae5af3f22929366b3d58a66dddb324153e76d2c/src/tscan.cxx#L3126-L3130

A connection is made with Frog, using this connection the plain text is send to Frog.

The text is then closed using `EOT` and Frog starts processing it. The response should then be read and written to `result`.

```cxx
client.write( "\nEOT\n" );
string result;
string s;
while ( client.read( s ) ) {
if ( s == "READY" )
break;
result += s + "\n";
}
```
> https://github.com/UUDigitalHumanitieslab/tscan/blob/1ae5af3f22929366b3d58a66dddb324153e76d2c/src/tscan.cxx#L3194C6-L3201

This will contain the FoliaXML in plain text which can then be loaded using

```cxx
doc = new folia::Document();
try {
doc->readFromString( result );
#ifdef DEBUG_FROG
cerr << "finished" << endl;
#endif
}
```
> https://github.com/UUDigitalHumanitieslab/tscan/blob/1ae5af3f22929366b3d58a66dddb324153e76d2c/src/tscan.cxx#L3210C1-L3216

The problem is however that the `result` is empty! Now this could be a problem with Frog.

So let's use Telnet:

```bash
$ telnet localhost 7001
Dit is een hele lang zin die niet gepakt wordt, dit is niet de zin, vraag mij om de zin en dan kan ik hem doorsturen.

EOT
```

And you get the response as expected:

```

READY

```

It may take a little while so it could perhaps be some race/timing issue.

I've tried giving it some more time e.g. adding a sleep before reading. I've also tried reading again after the initial attempt was empty after a sleep. Here's an attempt of me complicating the line ` while ( client.read( s ) ) {` into this:

```cxx
client.setNonBlocking();
// wait at least a little bit for the response
usleep( 100 );
while ( true ) {
if ( !client.read( s ) ) {
if ( !result.empty() ) {
// done reading
break;
}
else {
cerr << "giving Frog some more time... ";
sleep( 30 );
if ( !client.read( s ) ) {
// done reading
cerr << "still nothing" << endl;
break;
}
else {
cerr << s.size() << endl;
}
}
}
```

Anyway, it doesn't work or I wouldn't be writing this issue down. I'm going on leave for a month but I'm not quite sure how to fix this. _Possibly_ it's some bug in the socket implementation, maybe a work-around could be to simply use a different socket library. A super low tech (slow) approach could also be to call Frog from the command line which should also work, but is really slow because Frog will have to start up for every document. Maybe this could be moved to the Python service and let that deal with this. (Prepare all the input as FoliaXML files).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/tscan.cxx at the Frog connection around lines 3126-3130 and the response loop around lines 3194-3201; inspect how client.read() handles the response and READY marker. Reproduce the empty result with a complicated text and compare it with the Telnet response. Done means Frog output is reliably received and the FoliaXML is parsed into a document.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.