when running PROCEDURE HPNEURAL Sas stopped before reaching the assigned number of iteration value

Leona_2020 — Tue, 24 Nov 2020 16:09:42 GMT

Hi,

I got issues when I trained the model using HPNEURAL procedure. I have about 7 million records and 1200 columns dataset for training. parameter I set up 2 hidden layers each of them have 15 nodes, the max-iteration number is 1000. But it could not finish the model training process because of some error I could not understand. Please see the log:

proc hpneural data=&training.;
input &x_var / level = int;
target &y_var / level = int;
hidden 15;
hidden 15;
train outmodel=model numtries=1 maxiter=1000;
performance nodes=all details;
code file = "&path./NN1515_R_1000_&date..sas";
run;

NOTE: The HPNEURAL procedure is executing in the distributed computing environment with 3 worker nodes.
NOTE: Reading data...
NOTE: 6718589 usable observations in input data set.
NOTE: Training...
NOTE: Try 1 at iteration 1. Training Error=0.003217. Validation Error=0.003236
NOTE: Try 1 at iteration 3. Training Error=0.003037. Validation Error=0.003057
NOTE: Try 1 at iteration 5. Training Error=0.003032. Validation Error=0.003053
NOTE: Try 1 at iteration 7. Training Error=0.003004. Validation Error=0.003025
NOTE: Try 1 at iteration 9. Training Error=0.002896. Validation Error=0.002916
NOTE: Try 1 at iteration 10. Training Error=0.002848. Validation Error=0.002867
NOTE: Try 1 at iteration 12. Training Error=0.002830. Validation Error=0.002850
NOTE: Try 1 at iteration 14. Training Error=0.002813. Validation Error=0.002831
NOTE: Try 1 at iteration 16. Training Error=0.002811. Validation Error=0.002829
NOTE: Try 1 at iteration 18. Training Error=0.002791. Validation Error=0.002809
NOTE: Try 1 at iteration 21. Training Error=0.002747. Validation Error=0.002765
NOTE: Try 1 at iteration 23. Training Error=0.002725. Validation Error=0.002743
NOTE: Try 1 at iteration 24. Training Error=0.002723. Validation Error=0.002741
NOTE: Try 1 at iteration 26. Training Error=0.002719. Validation Error=0.002737
NOTE: Try 1 at iteration 28. Training Error=0.002717. Validation Error=0.002734
NOTE: Try 1 at iteration 30. Training Error=0.002705. Validation Error=0.002723
NOTE: Try 1 at iteration 32. Training Error=0.002676. Validation Error=0.002694
NOTE: Try 1 at iteration 34. Training Error=0.002656. Validation Error=0.002672
NOTE: Try 1 at iteration 36. Training Error=0.002649. Validation Error=0.002665
NOTE: Try 1 at iteration 38. Training Error=0.002627. Validation Error=0.002642
NOTE: Try 1 at iteration 40. Training Error=0.002610. Validation Error=0.002624
NOTE: Try 1 at iteration 42. Training Error=0.002604. Validation Error=0.002618
NOTE: Try 1 at iteration 44. Training Error=0.002603. Validation Error=0.002617
NOTE: Try 1 at iteration 46. Training Error=0.002595. Validation Error=0.002608
NOTE: Try 1 at iteration 48. Training Error=0.002586. Validation Error=0.002600
NOTE: Try 1 at iteration 50. Training Error=0.002581. Validation Error=0.002596
NOTE: Try 1 at iteration 52. Training Error=0.002577. Validation Error=0.002591
NOTE: Try 1 at iteration 53. Training Error=0.002576. Validation Error=0.002589
NOTE: Try 1 at iteration 54. Training Error=0.002574. Validation Error=0.002587
NOTE: Try 1 at iteration 56. Training Error=0.002570. Validation Error=0.002582
NOTE: Try 1 at iteration 57. Training Error=0.002569. Validation Error=0.002581
NOTE: Try 1 at iteration 59. Training Error=0.002568. Validation Error=0.002581
NOTE: Try 1 at iteration 61. Training Error=0.002568. Validation Error=0.002581
NOTE: Try 1 at iteration 63. Training Error=0.002567. Validation Error=0.002580
NOTE: Try 1 at iteration 65. Training Error=0.002565. Validation Error=0.002578
NOTE: Try 1 at iteration 68. Training Error=0.002560. Validation Error=0.002573
NOTE: Try 1 at iteration 69. Training Error=0.002559. Validation Error=0.002571
NOTE: Try 1 at iteration 71. Training Error=0.002554. Validation Error=0.002567
NOTE: Try 1 at iteration 73. Training Error=0.002552. Validation Error=0.002565
NOTE: Try 1 at iteration 75. Training Error=0.002550. Validation Error=0.002563
NOTE: Try 1 at iteration 77. Training Error=0.002548. Validation Error=0.002561
NOTE: Try 1 at iteration 79. Training Error=0.002544. Validation Error=0.002558
NOTE: Try 1 at iteration 82. Training Error=0.002542. Validation Error=0.002555
NOTE: Try 1 at iteration 85. Training Error=0.002540. Validation Error=0.002554
NOTE: Try 1 at iteration 88. Training Error=0.002538. Validation Error=0.002551
NOTE: Try 1 at iteration 92. Training Error=0.002535. Validation Error=0.002548
NOTE: Try 1 at iteration 95. Training Error=0.002534. Validation Error=0.002547
NOTE: Try 1 at iteration 98. Training Error=0.002533. Validation Error=0.002546
NOTE: Try 1 at iteration 101. Training Error=0.002533. Validation Error=0.002546
NOTE: Try 1 at iteration 104. Training Error=0.002531. Validation Error=0.002543
NOTE: Try 1 at iteration 107. Training Error=0.002526. Validation Error=0.002539
NOTE: Try 1 at iteration 111. Training Error=0.002524. Validation Error=0.002536
NOTE: Try 1 at iteration 115. Training Error=0.002522. Validation Error=0.002535
NOTE: Try 1 at iteration 119. Training Error=0.002517. Validation Error=0.002529
NOTE: Try 1 at iteration 122. Training Error=0.002516. Validation Error=0.002528
NOTE: Try 1 at iteration 125. Training Error=0.002513. Validation Error=0.002525
NOTE: Try 1 at iteration 128. Training Error=0.002510. Validation Error=0.002522
NOTE: Try 1 at iteration 130. Training Error=0.002509. Validation Error=0.002521
NOTE: Try 1 at iteration 133. Training Error=0.002507. Validation Error=0.002519
NOTE: Try 1 at iteration 136. Training Error=0.002503. Validation Error=0.002515
NOTE: Try 1 at iteration 138. Training Error=0.002501. Validation Error=0.002514
NOTE: Try 1 at iteration 141. Training Error=0.002500. Validation Error=0.002512
NOTE: Try 1 at iteration 145. Training Error=0.002499. Validation Error=0.002511
NOTE: Try 1 at iteration 148. Training Error=0.002498. Validation Error=0.002511
NOTE: Try 1 at iteration 152. Training Error=0.002497. Validation Error=0.002509
NOTE: Try 1 at iteration 156. Training Error=0.002496. Validation Error=0.002508
NOTE: Try 1 at iteration 159. Training Error=0.002496. Validation Error=0.002508
NOTE: Try 1 at iteration 162. Training Error=0.002494. Validation Error=0.002506
NOTE: Try 1 at iteration 164. Training Error=0.002493. Validation Error=0.002505
NOTE: Try 1 at iteration 166. Training Error=0.002493. Validation Error=0.002504
NOTE: Try 1 at iteration 169. Training Error=0.002490. Validation Error=0.002502
NOTE: Try 1 at iteration 172. Training Error=0.002488. Validation Error=0.002500
NOTE: Try 1 at iteration 176. Training Error=0.002487. Validation Error=0.002498
NOTE: Try 1 at iteration 178. Training Error=0.002486. Validation Error=0.002498
NOTE: Try 1 at iteration 181. Training Error=0.002486. Validation Error=0.002498
NOTE: Try 1 at iteration 184. Training Error=0.002485. Validation Error=0.002497
NOTE: Try 1 at iteration 187. Training Error=0.002485. Validation Error=0.002497
NOTE: Try 1 at iteration 190. Training Error=0.002484. Validation Error=0.002497
NOTE: Try 1 at iteration 193. Training Error=0.002482. Validation Error=0.002495
NOTE: Try 1 at iteration 196. Training Error=0.002481. Validation Error=0.002493
NOTE: Try 1 at iteration 199. Training Error=0.002480. Validation Error=0.002492
NOTE: Try 1 at iteration 202. Training Error=0.002479. Validation Error=0.002491
NOTE: Try 1 at iteration 204. Training Error=0.002479. Validation Error=0.002491
NOTE: Try 1 at iteration 207. Training Error=0.002477. Validation Error=0.002489
NOTE: Try 1 at iteration 210. Training Error=0.002476. Validation Error=0.002488
NOTE: Try 1 at iteration 212. Training Error=0.002476. Validation Error=0.002487
NOTE: Try 1 at iteration 215. Training Error=0.002475. Validation Error=0.002486
NOTE: Try 1 at iteration 218. Training Error=0.002474. Validation Error=0.002485
NOTE: Try 1 at iteration 221. Training Error=0.002473. Validation Error=0.002484
NOTE: Try 1 at iteration 226. Training Error=0.002472. Validation Error=0.002484
NOTE: Try 1 at iteration 231. Training Error=0.002470. Validation Error=0.002482
NOTE: Try 1 at iteration 235. Training Error=0.002469. Validation Error=0.002481
NOTE: Try 1 at iteration 240. Training Error=0.002467. Validation Error=0.002479
NOTE: Try 1 at iteration 244. Training Error=0.002467. Validation Error=0.002478
NOTE: Try 1 at iteration 248. Training Error=0.002464. Validation Error=0.002476
NOTE: Try 1 at iteration 252. Training Error=0.002463. Validation Error=0.002475
NOTE: Try 1 at iteration 255. Training Error=0.002462. Validation Error=0.002474
NOTE: Try 1 at iteration 259. Training Error=0.002460. Validation Error=0.002472
NOTE: Try 1 at iteration 262. Training Error=0.002459. Validation Error=0.002471
NOTE: Try 1 at iteration 267. Training Error=0.002456. Validation Error=0.002468
NOTE: Try 1 at iteration 272. Training Error=0.002455. Validation Error=0.002467
NOTE: Try 1 at iteration 277. Training Error=0.002454. Validation Error=0.002466
NOTE: Try 1 at iteration 281. Training Error=0.002454. Validation Error=0.002465
NOTE: Try 1 at iteration 286. Training Error=0.002451. Validation Error=0.002463
NOTE: Try 1 at iteration 290. Training Error=0.002450. Validation Error=0.002463
NOTE: Try 1 at iteration 293. Training Error=0.002448. Validation Error=0.002460
NOTE: Try 1 at iteration 297. Training Error=0.002447. Validation Error=0.002459
NOTE: Try 1 at iteration 300. Training Error=0.002446. Validation Error=0.002458
NOTE: Try 1 at iteration 305. Training Error=0.002444. Validation Error=0.002457
NOTE: Try 1 at iteration 309. Training Error=0.002444. Validation Error=0.002457
NOTE: Try 1 at iteration 313. Training Error=0.002442. Validation Error=0.002455
NOTE: Try 1 at iteration 316. Training Error=0.002441. Validation Error=0.002454
NOTE: Try 1 at iteration 319. Training Error=0.002441. Validation Error=0.002453
NOTE: Try 1 at iteration 322. Training Error=0.002441. Validation Error=0.002453
NOTE: Try 1 at iteration 325. Training Error=0.002440. Validation Error=0.002452
NOTE: Try 1 at iteration 328. Training Error=0.002439. Validation Error=0.002452
NOTE: Try 1 at iteration 331. Training Error=0.002437. Validation Error=0.002450
NOTE: Try 1 at iteration 333. Training Error=0.002437. Validation Error=0.002449
NOTE: Try 1 at iteration 336. Training Error=0.002437. Validation Error=0.002449
NOTE: Try 1 at iteration 338. Training Error=0.002436. Validation Error=0.002449
NOTE: Try 1 at iteration 341. Training Error=0.002436. Validation Error=0.002448
NOTE: Try 1 at iteration 343. Training Error=0.002435. Validation Error=0.002448
NOTE: Try 1 at iteration 346. Training Error=0.002434. Validation Error=0.002447
NOTE: Try 1 at iteration 349. Training Error=0.002434. Validation Error=0.002446
NOTE: Try 1 at iteration 351. Training Error=0.002434. Validation Error=0.002446
NOTE: Try 1 at iteration 354. Training Error=0.002433. Validation Error=0.002445
NOTE: Try 1 at iteration 357. Training Error=0.002432. Validation Error=0.002444
NOTE: Try 1 at iteration 360. Training Error=0.002431. Validation Error=0.002443
NOTE: Try 1 at iteration 363. Training Error=0.002431. Validation Error=0.002443
NOTE: Try 1 at iteration 366. Training Error=0.002431. Validation Error=0.002443
NOTE: Try 1 at iteration 368. Training Error=0.002431. Validation Error=0.002443
NOTE: Try 1 at iteration 370. Training Error=0.002430. Validation Error=0.002442
NOTE: Try 1 at iteration 372. Training Error=0.002429. Validation Error=0.002441
NOTE: Try 1 at iteration 375. Training Error=0.002429. Validation Error=0.002441
NOTE: Try 1 at iteration 377. Training Error=0.002428. Validation Error=0.002440
NOTE: Try 1 at iteration 380. Training Error=0.002428. Validation Error=0.002440
NOTE: Try 1 at iteration 383. Training Error=0.002427. Validation Error=0.002440
NOTE: Try 1 at iteration 386. Training Error=0.002427. Validation Error=0.002439
NOTE: Try 1 at iteration 389. Training Error=0.002426. Validation Error=0.002438
NOTE: Try 1 at iteration 391. Training Error=0.002425. Validation Error=0.002437
NOTE: Try 1 at iteration 394. Training Error=0.002424. Validation Error=0.002437
NOTE: Try 1 at iteration 397. Training Error=0.002424. Validation Error=0.002436
NOTE: Try 1 at iteration 400. Training Error=0.002423. Validation Error=0.002436
NOTE: Try 1 at iteration 403. Training Error=0.002423. Validation Error=0.002436
NOTE: Try 1 at iteration 406. Training Error=0.002423. Validation Error=0.002435
NOTE: Try 1 at iteration 408. Training Error=0.002422. Validation Error=0.002435
NOTE: Try 1 at iteration 411. Training Error=0.002421. Validation Error=0.002434
NOTE: Try 1 at iteration 414. Training Error=0.002421. Validation Error=0.002434
NOTE: The SAS System stopped processing this step because of errors.
NOTE: There were 6718589 observations read from the data set INT.MC_TRAIN_1106.
WARNING: The data set WORK.MODEL may be incomplete. When this step was stopped there were 0 observations and 0 variables.
NOTE: PROCEDURE HPNEURAL used (Total process time):
real time 4:00:15.21
user cpu time 1:42.49
system cpu time 59.64 seconds
memory 5814.65k
OS Memory 26372.00k
Timestamp 11/23/2020 08:14:29 PM
Step Count 2 Switch Count 2
Page Faults 9
Page Reclaims 2279
Page Swaps 0
Voluntary Context Switches 231262
Involuntary Context Switches 40057
Block Input Operations 1528
Block Output Operations 136

######################################

It seems like it decide to stop the process at some point for some error. but it did not tell details about error. In this case, I could not figure out how I could solve this issue. Hope anyone have met this issue before could help. Thanks

topic when running PROCEDURE HPNEURAL Sas stopped before reaching the assigned number of iteration value in SAS Enterprise Guide

when running PROCEDURE HPNEURAL Sas stopped before reaching the assigned number of iteration value