when running PROCEDURE HPNEURAL Sas stopped before reaching the assigned number of iteration value

[Leona_2020]

Hi,

I got issues when I trained the model using HPNEURAL procedure. I have about 7 million records and 1200 columns dataset for training. parameter I set up 2 hidden layers each of them have 15 nodes, the max-iteration number is 1000. But it could not finish the model training process because of some error I could not understand. Please see the log:

proc hpneural data=&training.;
 input &x_var / level = int;
 target &y_var / level = int;
 hidden 15;
 hidden 15;
train outmodel=model numtries=1 maxiter=1000;
performance nodes=all details;
code file = "&path./NN1515_R_1000_&date..sas";
 run;

NOTE: The HPNEURAL procedure is executing in the distributed computing environment with 3 worker nodes.
NOTE: Reading data...
NOTE: 6718589 usable observations in input data set.
NOTE: Training...
NOTE: Try 1 at iteration 1. Training Error=0.003217. Validation Error=0.003236
NOTE: Try 1 at iteration 3. Training Error=0.003037. Validation Error=0.003057
NOTE: Try 1 at iteration 5. Training Error=0.003032. Validation Error=0.003053
NOTE: Try 1 at iteration 7. Training Error=0.003004. Validation Error=0.003025
NOTE: Try 1 at iteration 9. Training Error=0.002896. Validation Error=0.002916
NOTE: Try 1 at iteration 10. Training Error=0.002848. Validation Error=0.002867
NOTE: Try 1 at iteration 12. Training Error=0.002830. Validation Error=0.002850
NOTE: Try 1 at iteration 14. Training Error=0.002813. Validation Error=0.002831
NOTE: Try 1 at iteration 16. Training Error=0.002811. Validation Error=0.002829
NOTE: Try 1 at iteration 18. Training Error=0.002791. Validation Error=0.002809
NOTE: Try 1 at iteration 21. Training Error=0.002747. Validation Error=0.002765
NOTE: Try 1 at iteration 23. Training Error=0.002725. Validation Error=0.002743
NOTE: Try 1 at iteration 24. Training Error=0.002723. Validation Error=0.002741
NOTE: Try 1 at iteration 26. Training Error=0.002719. Validation Error=0.002737
NOTE: Try 1 at iteration 28. Training Error=0.002717. Validation Error=0.002734
NOTE: Try 1 at iteration 30. Training Error=0.002705. Validation Error=0.002723
NOTE: Try 1 at iteration 32. Training Error=0.002676. Validation Error=0.002694
NOTE: Try 1 at iteration 34. Training Error=0.002656. Validation Error=0.002672
NOTE: Try 1 at iteration 36. Training Error=0.002649. Validation Error=0.002665
NOTE: Try 1 at iteration 38. Training Error=0.002627. Validation Error=0.002642
NOTE: Try 1 at iteration 40. Training Error=0.002610. Validation Error=0.002624
NOTE: Try 1 at iteration 42. Training Error=0.002604. Validation Error=0.002618
NOTE: Try 1 at iteration 44. Training Error=0.002603. Validation Error=0.002617
NOTE: Try 1 at iteration 46. Training Error=0.002595. Validation Error=0.002608
NOTE: Try 1 at iteration 48. Training Error=0.002586. Validation Error=0.002600
NOTE: Try 1 at iteration 50. Training Error=0.002581. Validation Error=0.002596
NOTE: Try 1 at iteration 52. Training Error=0.002577. Validation Error=0.002591
NOTE: Try 1 at iteration 53. Training Error=0.002576. Validation Error=0.002589
NOTE: Try 1 at iteration 54. Training Error=0.002574. Validation Error=0.002587
NOTE: Try 1 at iteration 56. Training Error=0.002570. Validation Error=0.002582
NOTE: Try 1 at iteration 57. Training Error=0.002569. Validation Error=0.002581
NOTE: Try 1 at iteration 59. Training Error=0.002568. Validation Error=0.002581
NOTE: Try 1 at iteration 61. Training Error=0.002568. Validation Error=0.002581
NOTE: Try 1 at iteration 63. Training Error=0.002567. Validation Error=0.002580
NOTE: Try 1 at iteration 65. Training Error=0.002565. Validation Error=0.002578
NOTE: Try 1 at iteration 68. Training Error=0.002560. Validation Error=0.002573
NOTE: Try 1 at iteration 69. Training Error=0.002559. Validation Error=0.002571
NOTE: Try 1 at iteration 71. Training Error=0.002554. Validation Error=0.002567
NOTE: Try 1 at iteration 73. Training Error=0.002552. Validation Error=0.002565
NOTE: Try 1 at iteration 75. Training Error=0.002550. Validation Error=0.002563
NOTE: Try 1 at iteration 77. Training Error=0.002548. Validation Error=0.002561
NOTE: Try 1 at iteration 79. Training Error=0.002544. Validation Error=0.002558
NOTE: Try 1 at iteration 82. Training Error=0.002542. Validation Error=0.002555
NOTE: Try 1 at iteration 85. Training Error=0.002540. Validation Error=0.002554
NOTE: Try 1 at iteration 88. Training Error=0.002538. Validation Error=0.002551
NOTE: Try 1 at iteration 92. Training Error=0.002535. Validation Error=0.002548
NOTE: Try 1 at iteration 95. Training Error=0.002534. Validation Error=0.002547
NOTE: Try 1 at iteration 98. Training Error=0.002533. Validation Error=0.002546
NOTE: Try 1 at iteration 101. Training Error=0.002533. Validation Error=0.002546
NOTE: Try 1 at iteration 104. Training Error=0.002531. Validation Error=0.002543
NOTE: Try 1 at iteration 107. Training Error=0.002526. Validation Error=0.002539
NOTE: Try 1 at iteration 111. Training Error=0.002524. Validation Error=0.002536
NOTE: Try 1 at iteration 115. Training Error=0.002522. Validation Error=0.002535
NOTE: Try 1 at iteration 119. Training Error=0.002517. Validation Error=0.002529
NOTE: Try 1 at iteration 122. Training Error=0.002516. Validation Error=0.002528
NOTE: Try 1 at iteration 125. Training Error=0.002513. Validation Error=0.002525
NOTE: Try 1 at iteration 128. Training Error=0.002510. Validation Error=0.002522
NOTE: Try 1 at iteration 130. Training Error=0.002509. Validation Error=0.002521
NOTE: Try 1 at iteration 133. Training Error=0.002507. Validation Error=0.002519
NOTE: Try 1 at iteration 136. Training Error=0.002503. Validation Error=0.002515
NOTE: Try 1 at iteration 138. Training Error=0.002501. Validation Error=0.002514
NOTE: Try 1 at iteration 141. Training Error=0.002500. Validation Error=0.002512
NOTE: Try 1 at iteration 145. Training Error=0.002499. Validation Error=0.002511
NOTE: Try 1 at iteration 148. Training Error=0.002498. Validation Error=0.002511
NOTE: Try 1 at iteration 152. Training Error=0.002497. Validation Error=0.002509
NOTE: Try 1 at iteration 156. Training Error=0.002496. Validation Error=0.002508
NOTE: Try 1 at iteration 159. Training Error=0.002496. Validation Error=0.002508
NOTE: Try 1 at iteration 162. Training Error=0.002494. Validation Error=0.002506
NOTE: Try 1 at iteration 164. Training Error=0.002493. Validation Error=0.002505
NOTE: Try 1 at iteration 166. Training Error=0.002493. Validation Error=0.002504
NOTE: Try 1 at iteration 169. Training Error=0.002490. Validation Error=0.002502
NOTE: Try 1 at iteration 172. Training Error=0.002488. Validation Error=0.002500
NOTE: Try 1 at iteration 176. Training Error=0.002487. Validation Error=0.002498
NOTE: Try 1 at iteration 178. Training Error=0.002486. Validation Error=0.002498
NOTE: Try 1 at iteration 181. Training Error=0.002486. Validation Error=0.002498
NOTE: Try 1 at iteration 184. Training Error=0.002485. Validation Error=0.002497
NOTE: Try 1 at iteration 187. Training Error=0.002485. Validation Error=0.002497
NOTE: Try 1 at iteration 190. Training Error=0.002484. Validation Error=0.002497
NOTE: Try 1 at iteration 193. Training Error=0.002482. Validation Error=0.002495
NOTE: Try 1 at iteration 196. Training Error=0.002481. Validation Error=0.002493
NOTE: Try 1 at iteration 199. Training Error=0.002480. Validation Error=0.002492
NOTE: Try 1 at iteration 202. Training Error=0.002479. Validation Error=0.002491
NOTE: Try 1 at iteration 204. Training Error=0.002479. Validation Error=0.002491
NOTE: Try 1 at iteration 207. Training Error=0.002477. Validation Error=0.002489
NOTE: Try 1 at iteration 210. Training Error=0.002476. Validation Error=0.002488
NOTE: Try 1 at iteration 212. Training Error=0.002476. Validation Error=0.002487
NOTE: Try 1 at iteration 215. Training Error=0.002475. Validation Error=0.002486
NOTE: Try 1 at iteration 218. Training Error=0.002474. Validation Error=0.002485
NOTE: Try 1 at iteration 221. Training Error=0.002473. Validation Error=0.002484
NOTE: Try 1 at iteration 226. Training Error=0.002472. Validation Error=0.002484
NOTE: Try 1 at iteration 231. Training Error=0.002470. Validation Error=0.002482
NOTE: Try 1 at iteration 235. Training Error=0.002469. Validation Error=0.002481
NOTE: Try 1 at iteration 240. Training Error=0.002467. Validation Error=0.002479
NOTE: Try 1 at iteration 244. Training Error=0.002467. Validation Error=0.002478
NOTE: Try 1 at iteration 248. Training Error=0.002464. Validation Error=0.002476
NOTE: Try 1 at iteration 252. Training Error=0.002463. Validation Error=0.002475
NOTE: Try 1 at iteration 255. Training Error=0.002462. Validation Error=0.002474
NOTE: Try 1 at iteration 259. Training Error=0.002460. Validation Error=0.002472
NOTE: Try 1 at iteration 262. Training Error=0.002459. Validation Error=0.002471
NOTE: Try 1 at iteration 267. Training Error=0.002456. Validation Error=0.002468
NOTE: Try 1 at iteration 272. Training Error=0.002455. Validation Error=0.002467
NOTE: Try 1 at iteration 277. Training Error=0.002454. Validation Error=0.002466
NOTE: Try 1 at iteration 281. Training Error=0.002454. Validation Error=0.002465
NOTE: Try 1 at iteration 286. Training Error=0.002451. Validation Error=0.002463
NOTE: Try 1 at iteration 290. Training Error=0.002450. Validation Error=0.002463
NOTE: Try 1 at iteration 293. Training Error=0.002448. Validation Error=0.002460
NOTE: Try 1 at iteration 297. Training Error=0.002447. Validation Error=0.002459
NOTE: Try 1 at iteration 300. Training Error=0.002446. Validation Error=0.002458
NOTE: Try 1 at iteration 305. Training Error=0.002444. Validation Error=0.002457
NOTE: Try 1 at iteration 309. Training Error=0.002444. Validation Error=0.002457
NOTE: Try 1 at iteration 313. Training Error=0.002442. Validation Error=0.002455
NOTE: Try 1 at iteration 316. Training Error=0.002441. Validation Error=0.002454
NOTE: Try 1 at iteration 319. Training Error=0.002441. Validation Error=0.002453
NOTE: Try 1 at iteration 322. Training Error=0.002441. Validation Error=0.002453
NOTE: Try 1 at iteration 325. Training Error=0.002440. Validation Error=0.002452
NOTE: Try 1 at iteration 328. Training Error=0.002439. Validation Error=0.002452
NOTE: Try 1 at iteration 331. Training Error=0.002437. Validation Error=0.002450
NOTE: Try 1 at iteration 333. Training Error=0.002437. Validation Error=0.002449
NOTE: Try 1 at iteration 336. Training Error=0.002437. Validation Error=0.002449
NOTE: Try 1 at iteration 338. Training Error=0.002436. Validation Error=0.002449
NOTE: Try 1 at iteration 341. Training Error=0.002436. Validation Error=0.002448
NOTE: Try 1 at iteration 343. Training Error=0.002435. Validation Error=0.002448
NOTE: Try 1 at iteration 346. Training Error=0.002434. Validation Error=0.002447
NOTE: Try 1 at iteration 349. Training Error=0.002434. Validation Error=0.002446
NOTE: Try 1 at iteration 351. Training Error=0.002434. Validation Error=0.002446
NOTE: Try 1 at iteration 354. Training Error=0.002433. Validation Error=0.002445
NOTE: Try 1 at iteration 357. Training Error=0.002432. Validation Error=0.002444
NOTE: Try 1 at iteration 360. Training Error=0.002431. Validation Error=0.002443
NOTE: Try 1 at iteration 363. Training Error=0.002431. Validation Error=0.002443
NOTE: Try 1 at iteration 366. Training Error=0.002431. Validation Error=0.002443
NOTE: Try 1 at iteration 368. Training Error=0.002431. Validation Error=0.002443
NOTE: Try 1 at iteration 370. Training Error=0.002430. Validation Error=0.002442
NOTE: Try 1 at iteration 372. Training Error=0.002429. Validation Error=0.002441
NOTE: Try 1 at iteration 375. Training Error=0.002429. Validation Error=0.002441
NOTE: Try 1 at iteration 377. Training Error=0.002428. Validation Error=0.002440
NOTE: Try 1 at iteration 380. Training Error=0.002428. Validation Error=0.002440
NOTE: Try 1 at iteration 383. Training Error=0.002427. Validation Error=0.002440
NOTE: Try 1 at iteration 386. Training Error=0.002427. Validation Error=0.002439
NOTE: Try 1 at iteration 389. Training Error=0.002426. Validation Error=0.002438
NOTE: Try 1 at iteration 391. Training Error=0.002425. Validation Error=0.002437
NOTE: Try 1 at iteration 394. Training Error=0.002424. Validation Error=0.002437
NOTE: Try 1 at iteration 397. Training Error=0.002424. Validation Error=0.002436
NOTE: Try 1 at iteration 400. Training Error=0.002423. Validation Error=0.002436
NOTE: Try 1 at iteration 403. Training Error=0.002423. Validation Error=0.002436
NOTE: Try 1 at iteration 406. Training Error=0.002423. Validation Error=0.002435
NOTE: Try 1 at iteration 408. Training Error=0.002422. Validation Error=0.002435
NOTE: Try 1 at iteration 411. Training Error=0.002421. Validation Error=0.002434
NOTE: Try 1 at iteration 414. Training Error=0.002421. Validation Error=0.002434
NOTE: The SAS System stopped processing this step because of errors.
NOTE: There were 6718589 observations read from the data set INT.MC_TRAIN_1106.
WARNING: The data set WORK.MODEL may be incomplete. When this step was stopped there were 0 observations and 0 variables.
NOTE: PROCEDURE HPNEURAL used (Total process time):
real time 4:00:15.21
user cpu time 1:42.49
system cpu time 59.64 seconds
memory 5814.65k
OS Memory 26372.00k
Timestamp 11/23/2020 08:14:29 PM
Step Count 2 Switch Count 2
Page Faults 9
Page Reclaims 2279
Page Swaps 0
Voluntary Context Switches 231262
Involuntary Context Switches 40057
Block Input Operations 1528
Block Output Operations 136

######################################

It seems like it decide to stop the process at some point for some error. but it did not tell details about error. In this case, I could not figure out how I could solve this issue. Hope anyone have met this issue before could help. Thanks