Hi, I got issues when I trained the model using HPNEURAL procedure. I have about 7 million records and 1200 columns dataset for training. parameter I set up 2 hidden layers each of them have 15 nodes, the max-iteration number is 1000. But it could not finish the model training process because of some error I could not understand. Please see the log: proc hpneural data=&training.; input &x_var / level = int; target &y_var / level = int; hidden 15; hidden 15; train outmodel=model numtries=1 maxiter=1000; performance nodes=all details; code file = "&path./NN1515_R_1000_&date..sas"; run; NOTE: The HPNEURAL procedure is executing in the distributed computing environment with 3 worker nodes. NOTE: Reading data... NOTE: 6718589 usable observations in input data set. NOTE: Training... NOTE: Try 1 at iteration 1. Training Error=0.003217. Validation Error=0.003236 NOTE: Try 1 at iteration 3. Training Error=0.003037. Validation Error=0.003057 NOTE: Try 1 at iteration 5. Training Error=0.003032. Validation Error=0.003053 NOTE: Try 1 at iteration 7. Training Error=0.003004. Validation Error=0.003025 NOTE: Try 1 at iteration 9. Training Error=0.002896. Validation Error=0.002916 NOTE: Try 1 at iteration 10. Training Error=0.002848. Validation Error=0.002867 NOTE: Try 1 at iteration 12. Training Error=0.002830. Validation Error=0.002850 NOTE: Try 1 at iteration 14. Training Error=0.002813. Validation Error=0.002831 NOTE: Try 1 at iteration 16. Training Error=0.002811. Validation Error=0.002829 NOTE: Try 1 at iteration 18. Training Error=0.002791. Validation Error=0.002809 NOTE: Try 1 at iteration 21. Training Error=0.002747. Validation Error=0.002765 NOTE: Try 1 at iteration 23. Training Error=0.002725. Validation Error=0.002743 NOTE: Try 1 at iteration 24. Training Error=0.002723. Validation Error=0.002741 NOTE: Try 1 at iteration 26. Training Error=0.002719. Validation Error=0.002737 NOTE: Try 1 at iteration 28. Training Error=0.002717. Validation Error=0.002734 NOTE: Try 1 at iteration 30. Training Error=0.002705. Validation Error=0.002723 NOTE: Try 1 at iteration 32. Training Error=0.002676. Validation Error=0.002694 NOTE: Try 1 at iteration 34. Training Error=0.002656. Validation Error=0.002672 NOTE: Try 1 at iteration 36. Training Error=0.002649. Validation Error=0.002665 NOTE: Try 1 at iteration 38. Training Error=0.002627. Validation Error=0.002642 NOTE: Try 1 at iteration 40. Training Error=0.002610. Validation Error=0.002624 NOTE: Try 1 at iteration 42. Training Error=0.002604. Validation Error=0.002618 NOTE: Try 1 at iteration 44. Training Error=0.002603. Validation Error=0.002617 NOTE: Try 1 at iteration 46. Training Error=0.002595. Validation Error=0.002608 NOTE: Try 1 at iteration 48. Training Error=0.002586. Validation Error=0.002600 NOTE: Try 1 at iteration 50. Training Error=0.002581. Validation Error=0.002596 NOTE: Try 1 at iteration 52. Training Error=0.002577. Validation Error=0.002591 NOTE: Try 1 at iteration 53. Training Error=0.002576. Validation Error=0.002589 NOTE: Try 1 at iteration 54. Training Error=0.002574. Validation Error=0.002587 NOTE: Try 1 at iteration 56. Training Error=0.002570. Validation Error=0.002582 NOTE: Try 1 at iteration 57. Training Error=0.002569. Validation Error=0.002581 NOTE: Try 1 at iteration 59. Training Error=0.002568. Validation Error=0.002581 NOTE: Try 1 at iteration 61. Training Error=0.002568. Validation Error=0.002581 NOTE: Try 1 at iteration 63. Training Error=0.002567. Validation Error=0.002580 NOTE: Try 1 at iteration 65. Training Error=0.002565. Validation Error=0.002578 NOTE: Try 1 at iteration 68. Training Error=0.002560. Validation Error=0.002573 NOTE: Try 1 at iteration 69. Training Error=0.002559. Validation Error=0.002571 NOTE: Try 1 at iteration 71. Training Error=0.002554. Validation Error=0.002567 NOTE: Try 1 at iteration 73. Training Error=0.002552. Validation Error=0.002565 NOTE: Try 1 at iteration 75. Training Error=0.002550. Validation Error=0.002563 NOTE: Try 1 at iteration 77. Training Error=0.002548. Validation Error=0.002561 NOTE: Try 1 at iteration 79. Training Error=0.002544. Validation Error=0.002558 NOTE: Try 1 at iteration 82. Training Error=0.002542. Validation Error=0.002555 NOTE: Try 1 at iteration 85. Training Error=0.002540. Validation Error=0.002554 NOTE: Try 1 at iteration 88. Training Error=0.002538. Validation Error=0.002551 NOTE: Try 1 at iteration 92. Training Error=0.002535. Validation Error=0.002548 NOTE: Try 1 at iteration 95. Training Error=0.002534. Validation Error=0.002547 NOTE: Try 1 at iteration 98. Training Error=0.002533. Validation Error=0.002546 NOTE: Try 1 at iteration 101. Training Error=0.002533. Validation Error=0.002546 NOTE: Try 1 at iteration 104. Training Error=0.002531. Validation Error=0.002543 NOTE: Try 1 at iteration 107. Training Error=0.002526. Validation Error=0.002539 NOTE: Try 1 at iteration 111. Training Error=0.002524. Validation Error=0.002536 NOTE: Try 1 at iteration 115. Training Error=0.002522. Validation Error=0.002535 NOTE: Try 1 at iteration 119. Training Error=0.002517. Validation Error=0.002529 NOTE: Try 1 at iteration 122. Training Error=0.002516. Validation Error=0.002528 NOTE: Try 1 at iteration 125. Training Error=0.002513. Validation Error=0.002525 NOTE: Try 1 at iteration 128. Training Error=0.002510. Validation Error=0.002522 NOTE: Try 1 at iteration 130. Training Error=0.002509. Validation Error=0.002521 NOTE: Try 1 at iteration 133. Training Error=0.002507. Validation Error=0.002519 NOTE: Try 1 at iteration 136. Training Error=0.002503. Validation Error=0.002515 NOTE: Try 1 at iteration 138. Training Error=0.002501. Validation Error=0.002514 NOTE: Try 1 at iteration 141. Training Error=0.002500. Validation Error=0.002512 NOTE: Try 1 at iteration 145. Training Error=0.002499. Validation Error=0.002511 NOTE: Try 1 at iteration 148. Training Error=0.002498. Validation Error=0.002511 NOTE: Try 1 at iteration 152. Training Error=0.002497. Validation Error=0.002509 NOTE: Try 1 at iteration 156. Training Error=0.002496. Validation Error=0.002508 NOTE: Try 1 at iteration 159. Training Error=0.002496. Validation Error=0.002508 NOTE: Try 1 at iteration 162. Training Error=0.002494. Validation Error=0.002506 NOTE: Try 1 at iteration 164. Training Error=0.002493. Validation Error=0.002505 NOTE: Try 1 at iteration 166. Training Error=0.002493. Validation Error=0.002504 NOTE: Try 1 at iteration 169. Training Error=0.002490. Validation Error=0.002502 NOTE: Try 1 at iteration 172. Training Error=0.002488. Validation Error=0.002500 NOTE: Try 1 at iteration 176. Training Error=0.002487. Validation Error=0.002498 NOTE: Try 1 at iteration 178. Training Error=0.002486. Validation Error=0.002498 NOTE: Try 1 at iteration 181. Training Error=0.002486. Validation Error=0.002498 NOTE: Try 1 at iteration 184. Training Error=0.002485. Validation Error=0.002497 NOTE: Try 1 at iteration 187. Training Error=0.002485. Validation Error=0.002497 NOTE: Try 1 at iteration 190. Training Error=0.002484. Validation Error=0.002497 NOTE: Try 1 at iteration 193. Training Error=0.002482. Validation Error=0.002495 NOTE: Try 1 at iteration 196. Training Error=0.002481. Validation Error=0.002493 NOTE: Try 1 at iteration 199. Training Error=0.002480. Validation Error=0.002492 NOTE: Try 1 at iteration 202. Training Error=0.002479. Validation Error=0.002491 NOTE: Try 1 at iteration 204. Training Error=0.002479. Validation Error=0.002491 NOTE: Try 1 at iteration 207. Training Error=0.002477. Validation Error=0.002489 NOTE: Try 1 at iteration 210. Training Error=0.002476. Validation Error=0.002488 NOTE: Try 1 at iteration 212. Training Error=0.002476. Validation Error=0.002487 NOTE: Try 1 at iteration 215. Training Error=0.002475. Validation Error=0.002486 NOTE: Try 1 at iteration 218. Training Error=0.002474. Validation Error=0.002485 NOTE: Try 1 at iteration 221. Training Error=0.002473. Validation Error=0.002484 NOTE: Try 1 at iteration 226. Training Error=0.002472. Validation Error=0.002484 NOTE: Try 1 at iteration 231. Training Error=0.002470. Validation Error=0.002482 NOTE: Try 1 at iteration 235. Training Error=0.002469. Validation Error=0.002481 NOTE: Try 1 at iteration 240. Training Error=0.002467. Validation Error=0.002479 NOTE: Try 1 at iteration 244. Training Error=0.002467. Validation Error=0.002478 NOTE: Try 1 at iteration 248. Training Error=0.002464. Validation Error=0.002476 NOTE: Try 1 at iteration 252. Training Error=0.002463. Validation Error=0.002475 NOTE: Try 1 at iteration 255. Training Error=0.002462. Validation Error=0.002474 NOTE: Try 1 at iteration 259. Training Error=0.002460. Validation Error=0.002472 NOTE: Try 1 at iteration 262. Training Error=0.002459. Validation Error=0.002471 NOTE: Try 1 at iteration 267. Training Error=0.002456. Validation Error=0.002468 NOTE: Try 1 at iteration 272. Training Error=0.002455. Validation Error=0.002467 NOTE: Try 1 at iteration 277. Training Error=0.002454. Validation Error=0.002466 NOTE: Try 1 at iteration 281. Training Error=0.002454. Validation Error=0.002465 NOTE: Try 1 at iteration 286. Training Error=0.002451. Validation Error=0.002463 NOTE: Try 1 at iteration 290. Training Error=0.002450. Validation Error=0.002463 NOTE: Try 1 at iteration 293. Training Error=0.002448. Validation Error=0.002460 NOTE: Try 1 at iteration 297. Training Error=0.002447. Validation Error=0.002459 NOTE: Try 1 at iteration 300. Training Error=0.002446. Validation Error=0.002458 NOTE: Try 1 at iteration 305. Training Error=0.002444. Validation Error=0.002457 NOTE: Try 1 at iteration 309. Training Error=0.002444. Validation Error=0.002457 NOTE: Try 1 at iteration 313. Training Error=0.002442. Validation Error=0.002455 NOTE: Try 1 at iteration 316. Training Error=0.002441. Validation Error=0.002454 NOTE: Try 1 at iteration 319. Training Error=0.002441. Validation Error=0.002453 NOTE: Try 1 at iteration 322. Training Error=0.002441. Validation Error=0.002453 NOTE: Try 1 at iteration 325. Training Error=0.002440. Validation Error=0.002452 NOTE: Try 1 at iteration 328. Training Error=0.002439. Validation Error=0.002452 NOTE: Try 1 at iteration 331. Training Error=0.002437. Validation Error=0.002450 NOTE: Try 1 at iteration 333. Training Error=0.002437. Validation Error=0.002449 NOTE: Try 1 at iteration 336. Training Error=0.002437. Validation Error=0.002449 NOTE: Try 1 at iteration 338. Training Error=0.002436. Validation Error=0.002449 NOTE: Try 1 at iteration 341. Training Error=0.002436. Validation Error=0.002448 NOTE: Try 1 at iteration 343. Training Error=0.002435. Validation Error=0.002448 NOTE: Try 1 at iteration 346. Training Error=0.002434. Validation Error=0.002447 NOTE: Try 1 at iteration 349. Training Error=0.002434. Validation Error=0.002446 NOTE: Try 1 at iteration 351. Training Error=0.002434. Validation Error=0.002446 NOTE: Try 1 at iteration 354. Training Error=0.002433. Validation Error=0.002445 NOTE: Try 1 at iteration 357. Training Error=0.002432. Validation Error=0.002444 NOTE: Try 1 at iteration 360. Training Error=0.002431. Validation Error=0.002443 NOTE: Try 1 at iteration 363. Training Error=0.002431. Validation Error=0.002443 NOTE: Try 1 at iteration 366. Training Error=0.002431. Validation Error=0.002443 NOTE: Try 1 at iteration 368. Training Error=0.002431. Validation Error=0.002443 NOTE: Try 1 at iteration 370. Training Error=0.002430. Validation Error=0.002442 NOTE: Try 1 at iteration 372. Training Error=0.002429. Validation Error=0.002441 NOTE: Try 1 at iteration 375. Training Error=0.002429. Validation Error=0.002441 NOTE: Try 1 at iteration 377. Training Error=0.002428. Validation Error=0.002440 NOTE: Try 1 at iteration 380. Training Error=0.002428. Validation Error=0.002440 NOTE: Try 1 at iteration 383. Training Error=0.002427. Validation Error=0.002440 NOTE: Try 1 at iteration 386. Training Error=0.002427. Validation Error=0.002439 NOTE: Try 1 at iteration 389. Training Error=0.002426. Validation Error=0.002438 NOTE: Try 1 at iteration 391. Training Error=0.002425. Validation Error=0.002437 NOTE: Try 1 at iteration 394. Training Error=0.002424. Validation Error=0.002437 NOTE: Try 1 at iteration 397. Training Error=0.002424. Validation Error=0.002436 NOTE: Try 1 at iteration 400. Training Error=0.002423. Validation Error=0.002436 NOTE: Try 1 at iteration 403. Training Error=0.002423. Validation Error=0.002436 NOTE: Try 1 at iteration 406. Training Error=0.002423. Validation Error=0.002435 NOTE: Try 1 at iteration 408. Training Error=0.002422. Validation Error=0.002435 NOTE: Try 1 at iteration 411. Training Error=0.002421. Validation Error=0.002434 NOTE: Try 1 at iteration 414. Training Error=0.002421. Validation Error=0.002434 NOTE: The SAS System stopped processing this step because of errors. NOTE: There were 6718589 observations read from the data set INT.MC_TRAIN_1106. WARNING: The data set WORK.MODEL may be incomplete. When this step was stopped there were 0 observations and 0 variables. NOTE: PROCEDURE HPNEURAL used (Total process time): real time 4:00:15.21 user cpu time 1:42.49 system cpu time 59.64 seconds memory 5814.65k OS Memory 26372.00k Timestamp 11/23/2020 08:14:29 PM Step Count 2 Switch Count 2 Page Faults 9 Page Reclaims 2279 Page Swaps 0 Voluntary Context Switches 231262 Involuntary Context Switches 40057 Block Input Operations 1528 Block Output Operations 136 ###################################### It seems like it decide to stop the process at some point for some error. but it did not tell details about error. In this case, I could not figure out how I could solve this issue. Hope anyone have met this issue before could help. Thanks
... View more