Failure on production machine

[sectorbets]

We run TWSLink on a production machine that hasn't seen any changes in months as well as on a development machine.

Today the production machine failed to make a couple of trades -- here is the log. It basically looks like there is a problem between the API and the IB algo we are using.

I want to emphasize that we have been using the same code unmodified for a couple of years and the same IB algo.

On the development machine, everything is running perfectly. The only difference between the two systems is that the production machine is running on a Windows 2012 server while the development machine is running on Windows 7.

This is a big problem, and if you can't spot the problem, we can rewrite our code to not use the IB algos, but that would be our last choice.

Thank you
Rick

20190327 09:45:00:116; 3212;note;tcExpFunc;PLACE_ORDER;-51192 -0 -BUY -MKT -867 -0.0000000000 -0.0000000000 -DAY --1 -0
20190327 09:45:00:117; 3212;note;tcExpFunc;PLACE_ORDER;ret=106353
20190327 09:45:00:118; 3212;note;tcExpFunc;SET_ORDER_VAL;-106353 -14 -0 --1
20190327 09:45:00:118; 3212;note;tcExpFunc;SET_ORDER_VAL;ret=1
20190327 09:45:00:118; 3212;note;tcExpFunc;SET_ORDER_VAL;-106353 -67 -Adaptive --1
20190327 09:45:00:118; 3212;note;tcExpFunc;SET_ORDER_VAL;ret=1
20190327 09:45:00:118; 3212;note;tcExpFunc;SET_ORDER_VAL;-106353 -68 -adaptivePriority=Normal --1
20190327 09:45:00:118; 3212;note;tcExpFunc;SET_ORDER_VAL;ret=1
20190327 09:45:00:118; 3212;note;tcExpFunc;SET_FA_MEMBERS;-106353 -acc_? - -SPY Algo Alloc - - 1
20190327 09:45:00:119; 4184;action;TWSSocket;FinishRequest;command [placeOrder] transmitted. Bytes sent=250
20190327 09:45:00:119; 4184;note;SmartOrder;openOrderCallExpected;flag=1 uid=106353
20190327 09:45:00:119 [OUT] uid=106353 BUY SPY MKT 867 DAY SMART lmt=0 aux=0 Order in Transit acc_? clientid=1 transmit=IB ref= nref= oca= filled=0 lastfill=0 fa group= fa profile=SPY Algo Alloc fa method= fa percentage= gat= gtd= outside rth=0 hidden=0 algo=Adaptive algo-param=adaptivePriority=Normal oid=3170 permid=0 uidp=0 oidp=0 cid=51192 comment=
20190327 09:45:00:154; 3212;note;tcExpFunc;SET_FA_MEMBERS;ret=1
20190327 09:45:00:241; 3212;note;tcExpFunc;PLACE_ORDER;-51194 -0 -BUY -MKT -2349 -0.0000000000 -0.0000000000 -DAY --1 -0
20190327 09:45:00:245; 3212;note;tcExpFunc;PLACE_ORDER;ret=106354
20190327 09:45:00:245; 3212;note;tcExpFunc;SET_ORDER_VAL;-106354 -14 -0 --1
20190327 09:45:00:245; 3212;note;tcExpFunc;SET_ORDER_VAL;ret=1
20190327 09:45:00:245; 3212;note;tcExpFunc;SET_ORDER_VAL;-106354 -67 -Adaptive --1
20190327 09:45:00:245; 3212;note;tcExpFunc;SET_ORDER_VAL;ret=1
20190327 09:45:00:245; 3212;note;tcExpFunc;SET_ORDER_VAL;-106354 -68 -adaptivePriority=Normal --1
20190327 09:45:00:245; 3212;note;tcExpFunc;SET_ORDER_VAL;ret=1
20190327 09:45:00:246; 3212;note;tcExpFunc;SET_FA_MEMBERS;-106354 -acc_? - -SSO SDS 2X Alloc - - 1
20190327 09:45:00:246; 4184;action;TWSSocket;FinishRequest;command [placeOrder] transmitted. Bytes sent=256
20190327 09:45:00:246; 4184;note;SmartOrder;openOrderCallExpected;flag=1 uid=106354
20190327 09:45:00:246 [OUT] uid=106354 BUY SSO MKT 2349 DAY SMART lmt=0 aux=0 Order in Transit acc_? clientid=1 transmit=IB ref= nref= oca= filled=0 lastfill=0 fa group= fa profile=SSO SDS 2X Alloc fa method= fa percentage= gat= gtd= outside rth=0 hidden=0 algo=Adaptive algo-param=adaptivePriority=Normal oid=3171 permid=0 uidp=0 oidp=0 cid=51194 comment=
20190327 09:45:00:247; 3212;note;tcExpFunc;SET_FA_MEMBERS;ret=1
20190327 09:45:00:471; 4184;note;TWSSocket;position;postions of contract SPY_STK_USD___0.000000_=14 conid=756733 account=acc_35 context=updatePortfolio avg=229.6864285 calls=4 publish_pos=1 publish_pl=1
20190327 09:45:00:472; 4184;note;TWSSocket;position;postions of contract SPY_STK_USD___0.000000_=3 conid=756733 account=acc_10 context=updatePortfolio avg=128.2743666 calls=4 publish_pos=1 publish_pl=1
20190327 09:45:00:472; 4184;note;libwts_db;perf;perf(20-avg,update portfolio)=00:00:00:00:000:358
20190327 09:45:00:472; 4184;note;TWSSocket;position;postions of contract SPY_STK_USD___0.000000_=11942 conid=756733 account=acc_29 context=updatePortfolio avg=90.0185606 calls=4 publish_pos=1 publish_pl=1
20190327 09:45:00:473; 4184;note;TWSSocket;position;postions of contract SSO_STK_USD___0.000000_=21 conid=39622943 account=acc_34 context=updatePortfolio avg=113.9251666 calls=4 publish_pos=1 publish_pl=1
20190327 09:45:00:474; 4184;note;TWSSocket;position;postions of contract SSO_STK_USD___0.000000_=80000 conid=39622943 account=acc_29 context=updatePortfolio avg=11.1481320 calls=4 publish_pos=1 publish_pl=1
20190327 09:45:00:555; 4184;error;API;;API-error: specified algorithm is not allowed for this order. code=442 id=3170
20190327 09:45:00:557; 4184;warning;API;error;unknown error message so far: code=442 message=Specified algorithm is not allowed for this order.
20190327 09:45:00:557; 4184;note;SmartOrder;openOrderCallExpected;flag=0 uid=106353
20190327 09:45:00:557; 4184;error;API;;API-error: specified algorithm is not allowed for this order. code=442 id=3171
20190327 09:45:00:557; 4184;warning;API;error;unknown error message so far: code=442 message=Specified algorithm is not allowed for this order.

[board_admin]

Hello,
the last line seems to point to the problem:

Specified algorithm is not allowed for this order

[sectorbets]

There has been an additional trade (VXXB) and it failed with the same messages about the algo not allowed.

[sectorbets]

There has been an additional trade (VXXB) and it failed with the same messages about the algo not allowed.

[sectorbets]

And yet another trade failed in the same manor -- let me remind you that exactly the same code is running on our development machine and it is working correctly. The production code has been running for months with no issues, including just yesterday, so we don't know where to look.

[board_admin]

This message comes form IB. I don't believe it is a TWSLink problem.

You either missed to pass a paramter or the parameter you passed
are not valid. This can be related to account, precautions...

Please contact IB support about this.
First make sure to have the logfiles.
In the API settings of TWS, enable API log and logging level at least to info.

Try to reproduce the error, collect logs and send them to IB support.
Give feedback please about results.

Thank you

[sectorbets]

We will contact IB, of course, but I want to emphasize that the machine that is failing has code that has been locked down for many months, so there are only three possibilities-- the API interface has changed, server 2012 has changed, or your code has changed.

I think we can assume your code has not changed overnight since you are no longer adding features and are only providing maintenance when absolutely necessary. I spoke to our data center and they have made no recent changes to server 2012 installation, and I will speak to IB as soon as I can get them.

We had a similar problem a few months ago when the combination of server 2012, your code, and an API change combined to render our system unusable. You at that time determined that you were not going to dig into the server 2012 installation since you were running perfectly on Windows 7/10, so we fell back to an earlier version of TWS that works perfectly on server 2012 until today. I suspect there was another change to the API rendering server 2012 inoperable again, but unless you can dig into it, the only real solution is to turn off the IB algos to see what happens.

[sectorbets]

It appears that other TradeStation customers are having a similar problem according to IB. They have isolated the problem to users on their Greenwich servers -- our failing system is on those servers. They are not seeing the problem on their other servers, and as it turns out, our other systems are on other IB servers, so no problem. They hope to have a solution by tomorrow/.

[board_admin]

Thank you, please keep us updated

[sectorbets]

The problem appears to be fixed -- we had a successful trade transmitted from TradeStation to IB just before the close --