Improbable Icon

SpatialOS Discourse Forums

Unreal Engine Fatal Error after Join

Hi guys,

I’ve been experiencing a fatal error crash after joining a game, the terrain on my map I am using is rather large. The timing of the crash varies from session to session, but it doesn’t take long to happen.
fatal_crash

Looking at the logs on the deployment’s dashboard doesn’t show any errors except for the following:

Looking at the launcher logs, I get the following (project name replaced with “xxxxx”, windows user name changed to “windowsUser”):

{“message”:"‘version: 1.10.0 channel: STABLE’",“time”:“Mon, 02 Dec 2019 03:53:35 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘reading C:\\Users\\windowsUser\\AppData\\Local\\SpatialOS\\Launch\\cache_log.json’",“time”:“Mon, 02 Dec 2019 03:53:39 GMT”,“level”:“INFO”,“name”:“cache-log”}
{“message”:"‘in log: 8 items, in cache: 8 items, retained (intersection): 8’",“time”:“Mon, 02 Dec 2019 03:53:39 GMT”,“level”:“DEBUG”,“name”:“cache-log”}
{“message”:"‘wrote log with 8 item(s) to C:\\Users\\windowsUser\\AppData\\Local\\SpatialOS\\Launch\\cache_log.json’",“time”:“Mon, 02 Dec 2019 03:53:39 GMT”,“level”:“INFO”,“name”:“cache-log”}
{“message”:"‘instantiated from file C:\\Users\\windowsUser\\AppData\\Local\\SpatialOS\\Launch\\cache_log.json and directory C:\\Users\\windowsUser\\AppData\\Local\\SpatialOS\\Launch\\clients, old size: 8, new size: 8’",“time”:“Mon, 02 Dec 2019 03:53:39 GMT”,“level”:“DEBUG”,“name”:“cache-log”}
{“message”:"‘Settings user tasks’ [ { program: ‘C:\\Program Files\\Improbable\\SpatialOS Launcher\\SpatialOS Launcher.exe’,\n arguments: ‘spatialos.launch:beta_blue_robert_270-xxxxxxprototypedeployment?token=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRwbCI6IjMwODAyMDI0NDY1MTMxMTQ0NjQiLCJ3a3QiOiIiLCJpYXQiOjE1NzUyNTg4MTEsImV4cCI6MTU3NTI2MDYxMX0.hA4bpXNwdtA0_5q0I80HLKocD3_AdRwIosaaTLU5cHRdVACHX_-cPaqmO6AtYUbRkhPFBBE_zt3CAXSOIeMsfSf4LfgGkryFBSSC-s65Saqq7Ywkd0oQfbFKGDvat7BvvaeJsqApyFMzmeSdLFTBZGUI7_0ZmB-DwP8SNXZ9O8APQ4OB0mVLuI8lxKXcA6XS6f5N_aVqiF2W8GoQAy4IH0ChZkadW68tx2RMG8qt2B23tiqYLJhWea4USd3B83UsKHWpSEhERY2H01CfCa134BTqOu0lUGzNFAHUSBg8N4GDymmpptLLU_k2grB2yEHX3KrjMlbRwRA8ddzTJbo3sQ&asm=xxxxxxprototypeassembly&pit=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRuIjoiUnlhbiBUaG9ybnRvbiIsImlhdCI6IjIwMTktMTItMDJUMDM6NTM6MzEuMDQ5MjQyMzg0WiIsImV4cCI6IjIwMTktMTItMDNUMDM6NTM6MzEuMDQ5MjQyMzg0WiJ9.F5YofziTBUB2ZSEJOhYPF8yzxeVPvohTUD7oJbCjUxt5hjIK-OIP7apjgpf2-pN4bhx1_BgYtpU7o90kMIH5UOmcIpQhdfUx4Bjd62SJ4_vvzl7SBNWYeGHWneZJplZSqD0oMJdzvWohuehvsruimm7JaAqMYtBhjGzeaCbQbm7BJEpymltuk1wvRY_dpg7n37Xx0Txdc68lGqIhDUadnFeDK2SMKyGNJ78xAIYbfLbDBHIPxlEPDTjiM-vocJKQ_IXg4qGC2RY_vQWwoWrfTJC2QHMbkfPcqlJ0bH-kx6auge9PMWBT4Ve5zqnxkeszycyyIWAGjACThaCBj_imFw --clear-cache’,\n title: ‘Clear cache’,\n description: ‘Clears the cache’,\n iconPath: ‘C:\\Program Files\\Improbable\\SpatialOS Launcher\\SpatialOS Launcher.exe’,\n iconIndex: 0 },\n { program: ‘C:\\Program Files\\Improbable\\SpatialOS Launcher\\SpatialOS Launcher.exe’,\n arguments: ‘spatialos.launch:beta_blue_robert_270-xxxxxxprototypedeployment?token=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRwbCI6IjMwODAyMDI0NDY1MTMxMTQ0NjQiLCJ3a3QiOiIiLCJpYXQiOjE1NzUyNTg4MTEsImV4cCI6MTU3NTI2MDYxMX0.hA4bpXNwdtA0_5q0I80HLKocD3_AdRwIosaaTLU5cHRdVACHX_-cPaqmO6AtYUbRkhPFBBE_zt3CAXSOIeMsfSf4LfgGkryFBSSC-s65Saqq7Ywkd0oQfbFKGDvat7BvvaeJsqApyFMzmeSdLFTBZGUI7_0ZmB-DwP8SNXZ9O8APQ4OB0mVLuI8lxKXcA6XS6f5N_aVqiF2W8GoQAy4IH0ChZkadW68tx2RMG8qt2B23tiqYLJhWea4USd3B83UsKHWpSEhERY2H01CfCa134BTqOu0lUGzNFAHUSBg8N4GDymmpptLLU_k2grB2yEHX3KrjMlbRwRA8ddzTJbo3sQ&asm=xxxxxxprototypeassembly&pit=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRuIjoiUnlhbiBUaG9ybnRvbiIsImlhdCI6IjIwMTktMTItMDJUMDM6NTM6MzEuMDQ5MjQyMzg0WiIsImV4cCI6IjIwMTktMTItMDNUMDM6NTM6MzEuMDQ5MjQyMzg0WiJ9.F5YofziTBUB2ZSEJOhYPF8yzxeVPvohTUD7oJbCjUxt5hjIK-OIP7apjgpf2-pN4bhx1_BgYtpU7o90kMIH5UOmcIpQhdfUx4Bjd62SJ4_vvzl7SBNWYeGHWneZJplZSqD0oMJdzvWohuehvsruimm7JaAqMYtBhjGzeaCbQbm7BJEpymltuk1wvRY_dpg7n37Xx0Txdc68lGqIhDUadnFeDK2SMKyGNJ78xAIYbfLbDBHIPxlEPDTjiM-vocJKQ_IXg4qGC2RY_vQWwoWrfTJC2QHMbkfPcqlJ0bH-kx6auge9PMWBT4Ve5zqnxkeszycyyIWAGjACThaCBj_imFw --show-logs’,\n title: ‘Show Logs’,\n description: ‘Opens up the locations of the log files in the system file explorer’,\n iconPath: ‘C:\\Program Files\\Improbable\\SpatialOS Launcher\\SpatialOS Launcher.exe’,\n iconIndex: 0 } ]",“time”:“Mon, 02 Dec 2019 03:53:39 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘executed 0 launcher actions’",“time”:“Mon, 02 Dec 2019 03:53:40 GMT”,“level”:“DEBUG”,“name”:“launcher”}
{“message”:"‘ready’",“time”:“Mon, 02 Dec 2019 03:53:40 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘Launch URL: spatialos.launch:beta_blue_robert_270-xxxxxxprototypedeployment?token=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRwbCI6IjMwODAyMDI0NDY1MTMxMTQ0NjQiLCJ3a3QiOiIiLCJpYXQiOjE1NzUyNTg4MTEsImV4cCI6MTU3NTI2MDYxMX0.hA4bpXNwdtA0_5q0I80HLKocD3_AdRwIosaaTLU5cHRdVACHX_-cPaqmO6AtYUbRkhPFBBE_zt3CAXSOIeMsfSf4LfgGkryFBSSC-s65Saqq7Ywkd0oQfbFKGDvat7BvvaeJsqApyFMzmeSdLFTBZGUI7_0ZmB-DwP8SNXZ9O8APQ4OB0mVLuI8lxKXcA6XS6f5N_aVqiF2W8GoQAy4IH0ChZkadW68tx2RMG8qt2B23tiqYLJhWea4USd3B83UsKHWpSEhERY2H01CfCa134BTqOu0lUGzNFAHUSBg8N4GDymmpptLLU_k2grB2yEHX3KrjMlbRwRA8ddzTJbo3sQ&asm=xxxxxxprototypeassembly&pit=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRuIjoiUnlhbiBUaG9ybnRvbiIsImlhdCI6IjIwMTktMTItMDJUMDM6NTM6MzEuMDQ5MjQyMzg0WiIsImV4cCI6IjIwMTktMTItMDNUMDM6NTM6MzEuMDQ5MjQyMzg0WiJ9.F5YofziTBUB2ZSEJOhYPF8yzxeVPvohTUD7oJbCjUxt5hjIK-OIP7apjgpf2-pN4bhx1_BgYtpU7o90kMIH5UOmcIpQhdfUx4Bjd62SJ4_vvzl7SBNWYeGHWneZJplZSqD0oMJdzvWohuehvsruimm7JaAqMYtBhjGzeaCbQbm7BJEpymltuk1wvRY_dpg7n37Xx0Txdc68lGqIhDUadnFeDK2SMKyGNJ78xAIYbfLbDBHIPxlEPDTjiM-vocJKQ_IXg4qGC2RY_vQWwoWrfTJC2QHMbkfPcqlJ0bH-kx6auge9PMWBT4Ve5zqnxkeszycyyIWAGjACThaCBj_imFw. Program arguments C:\\Program Files\\Improbable\\SpatialOS Launcher\\SpatialOS Launcher.exe, spatialos.launch:beta_blue_robert_270-xxxxxxprototypedeployment?token=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRwbCI6IjMwODAyMDI0NDY1MTMxMTQ0NjQiLCJ3a3QiOiIiLCJpYXQiOjE1NzUyNTg4MTEsImV4cCI6MTU3NTI2MDYxMX0.hA4bpXNwdtA0_5q0I80HLKocD3_AdRwIosaaTLU5cHRdVACHX_-cPaqmO6AtYUbRkhPFBBE_zt3CAXSOIeMsfSf4LfgGkryFBSSC-s65Saqq7Ywkd0oQfbFKGDvat7BvvaeJsqApyFMzmeSdLFTBZGUI7_0ZmB-DwP8SNXZ9O8APQ4OB0mVLuI8lxKXcA6XS6f5N_aVqiF2W8GoQAy4IH0ChZkadW68tx2RMG8qt2B23tiqYLJhWea4USd3B83UsKHWpSEhERY2H01CfCa134BTqOu0lUGzNFAHUSBg8N4GDymmpptLLU_k2grB2yEHX3KrjMlbRwRA8ddzTJbo3sQ&asm=xxxxxxprototypeassembly&pit=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRuIjoiUnlhbiBUaG9ybnRvbiIsImlhdCI6IjIwMTktMTItMDJUMDM6NTM6MzEuMDQ5MjQyMzg0WiIsImV4cCI6IjIwMTktMTItMDNUMDM6NTM6MzEuMDQ5MjQyMzg0WiJ9.F5YofziTBUB2ZSEJOhYPF8yzxeVPvohTUD7oJbCjUxt5hjIK-OIP7apjgpf2-pN4bhx1_BgYtpU7o90kMIH5UOmcIpQhdfUx4Bjd62SJ4_vvzl7SBNWYeGHWneZJplZSqD0oMJdzvWohuehvsruimm7JaAqMYtBhjGzeaCbQbm7BJEpymltuk1wvRY_dpg7n37Xx0Txdc68lGqIhDUadnFeDK2SMKyGNJ78xAIYbfLbDBHIPxlEPDTjiM-vocJKQ_IXg4qGC2RY_vQWwoWrfTJC2QHMbkfPcqlJ0bH-kx6auge9PMWBT4Ve5zqnxkeszycyyIWAGjACThaCBj_imFw’",“time”:“Mon, 02 Dec 2019 03:53:40 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"'onReady with url: spatialos.launch:beta_blue_robert_270-xxxxxxprototypedeployment?token=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRwbCI6IjMwODAyMDI0NDY1MTMxMTQ0NjQiLCJ3a3QiOiIiLCJpYXQiOjE1NzUyNTg4MTEsImV4cCI6MTU3NTI2MDYxMX0.hA4bpXNwdtA0_5q0I80HLKocD3_AdRwIosaaTLU5cHRdVACHX_-cPaqmO6AtYUbRkhPFBBE_zt3CAXSOIeMsfSf4LfgGkryFBSSC-s65Saqq7Ywkd0oQfbFKGDvat7BvvaeJsqApyFMzmeSdLFTBZGUI7_0ZmB-DwP8SNXZ9O8APQ4OB0mVLuI8lxKXcA6XS6f5N_aVqiF2W8GoQAy4IH0ChZkadW68tx2RMG8qt2B23tiqYLJhWea4USd3B83UsKHWpSEhERY2H01CfCa134BTqOu0lUGzNFAHUSBg8N4GDymmpptLLU_k2grB2yEHX3KrjMlbRwRA8ddzTJbo3sQ&asm=xxxxxxprototypeassembly&pit=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRuIjoiUnlhbiBUaG9ybnRvbiIsImlhdCI6IjIwMTktMTItMDJUMDM6NTM6MzEuMDQ5MjQyMzg0WiIsImV4cCI6IjIwMTktMTItMDNUMDM6NTM6MzEuMDQ5MjQyMzg0WiJ9.F5YofziTBUB2ZSEJOhYPF8yzxeVPvohTUD7oJbCjUxt5hjIK-OIP7apjgpf2-pN4bhx1_BgYtpU7o90kMIH5UOmcIpQhdfUx4Bjd62SJ4_vvzl7SBNWYeGHWneZJplZSqD0oMJdzvWohuehvsruimm7JaAqMYtBhjGzeaCbQbm7BJEpymltuk1wvRY_dpg7n37Xx0Txdc68lGqIhDUadnFeDK2SMKyGNJ78xAIYbfLbDBHIPxlEPDTjiM-vocJKQ_IXg4qGC2RY_vQWwoWrfTJC2QHMbkfPcqlJ0bH-kx6auge9PMWBT4Ve5zqnxkeszycyyIWAGjACThaCBj_imFw and args '",“time”:“Mon, 02 Dec 2019 03:53:40 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘loading html from: C:\\Program Files\\Improbable\\SpatialOS Launcher\\resources\\app.asar\\src\\index.html’",“time”:“Mon, 02 Dec 2019 03:53:40 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘parsing login token: spatialos.launch:beta_blue_robert_270-xxxxxxprototypedeployment?token=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRwbCI6IjMwODAyMDI0NDY1MTMxMTQ0NjQiLCJ3a3QiOiIiLCJpYXQiOjE1NzUyNTg4MTEsImV4cCI6MTU3NTI2MDYxMX0.hA4bpXNwdtA0_5q0I80HLKocD3_AdRwIosaaTLU5cHRdVACHX_-cPaqmO6AtYUbRkhPFBBE_zt3CAXSOIeMsfSf4LfgGkryFBSSC-s65Saqq7Ywkd0oQfbFKGDvat7BvvaeJsqApyFMzmeSdLFTBZGUI7_0ZmB-DwP8SNXZ9O8APQ4OB0mVLuI8lxKXcA6XS6f5N_aVqiF2W8GoQAy4IH0ChZkadW68tx2RMG8qt2B23tiqYLJhWea4USd3B83UsKHWpSEhERY2H01CfCa134BTqOu0lUGzNFAHUSBg8N4GDymmpptLLU_k2grB2yEHX3KrjMlbRwRA8ddzTJbo3sQ&asm=xxxxxxprototypeassembly&pit=eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRuIjoiUnlhbiBUaG9ybnRvbiIsImlhdCI6IjIwMTktMTItMDJUMDM6NTM6MzEuMDQ5MjQyMzg0WiIsImV4cCI6IjIwMTktMTItMDNUMDM6NTM6MzEuMDQ5MjQyMzg0WiJ9.F5YofziTBUB2ZSEJOhYPF8yzxeVPvohTUD7oJbCjUxt5hjIK-OIP7apjgpf2-pN4bhx1_BgYtpU7o90kMIH5UOmcIpQhdfUx4Bjd62SJ4_vvzl7SBNWYeGHWneZJplZSqD0oMJdzvWohuehvsruimm7JaAqMYtBhjGzeaCbQbm7BJEpymltuk1wvRY_dpg7n37Xx0Txdc68lGqIhDUadnFeDK2SMKyGNJ78xAIYbfLbDBHIPxlEPDTjiM-vocJKQ_IXg4qGC2RY_vQWwoWrfTJC2QHMbkfPcqlJ0bH-kx6auge9PMWBT4Ve5zqnxkeszycyyIWAGjACThaCBj_imFw’",“time”:“Mon, 02 Dec 2019 03:53:40 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘launching from url: {“projectName”:“beta_blue_robert_270”,“deploymentName”:“xxxxxxprototypedeployment”,“token”:“eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRwbCI6IjMwODAyMDI0NDY1MTMxMTQ0NjQiLCJ3a3QiOiIiLCJpYXQiOjE1NzUyNTg4MTEsImV4cCI6MTU3NTI2MDYxMX0.hA4bpXNwdtA0_5q0I80HLKocD3_AdRwIosaaTLU5cHRdVACHX_-cPaqmO6AtYUbRkhPFBBE_zt3CAXSOIeMsfSf4LfgGkryFBSSC-s65Saqq7Ywkd0oQfbFKGDvat7BvvaeJsqApyFMzmeSdLFTBZGUI7_0ZmB-DwP8SNXZ9O8APQ4OB0mVLuI8lxKXcA6XS6f5N_aVqiF2W8GoQAy4IH0ChZkadW68tx2RMG8qt2B23tiqYLJhWea4USd3B83UsKHWpSEhERY2H01CfCa134BTqOu0lUGzNFAHUSBg8N4GDymmpptLLU_k2grB2yEHX3KrjMlbRwRA8ddzTJbo3sQ”,“playerIdentityToken”:“eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRuIjoiUnlhbiBUaG9ybnRvbiIsImlhdCI6IjIwMTktMTItMDJUMDM6NTM6MzEuMDQ5MjQyMzg0WiIsImV4cCI6IjIwMTktMTItMDNUMDM6NTM6MzEuMDQ5MjQyMzg0WiJ9.F5YofziTBUB2ZSEJOhYPF8yzxeVPvohTUD7oJbCjUxt5hjIK-OIP7apjgpf2-pN4bhx1_BgYtpU7o90kMIH5UOmcIpQhdfUx4Bjd62SJ4_vvzl7SBNWYeGHWneZJplZSqD0oMJdzvWohuehvsruimm7JaAqMYtBhjGzeaCbQbm7BJEpymltuk1wvRY_dpg7n37Xx0Txdc68lGqIhDUadnFeDK2SMKyGNJ78xAIYbfLbDBHIPxlEPDTjiM-vocJKQ_IXg4qGC2RY_vQWwoWrfTJC2QHMbkfPcqlJ0bH-kx6auge9PMWBT4Ve5zqnxkeszycyyIWAGjACThaCBj_imFw”,“assemblyName”:“xxxxxxprototypeassembly”}’",“time”:“Mon, 02 Dec 2019 03:53:40 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘checking for update from url: https://console.improbable.io/launcher/check-update?version=1.10.0&platform=win&channel=STABLE’",“time”:“Mon, 02 Dec 2019 03:53:40 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘no update available. status code: 204’",“time”:“Mon, 02 Dec 2019 03:53:42 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘no update - user dismissed: false’",“time”:“Mon, 02 Dec 2019 03:53:42 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘sending to window: no-update’",“time”:“Mon, 02 Dec 2019 03:53:42 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘sending to window: project-name’",“time”:“Mon, 02 Dec 2019 03:53:42 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘Preparing to retrieve client url for assembly \‘xxxxxxprototypeassembly\’ from https://clients.service.improbable.io’",“time”:“Mon, 02 Dec 2019 03:53:43 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘Fetching client url from https://clients.service.improbable.io’",“time”:“Mon, 02 Dec 2019 03:53:43 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘Attempting to retrieve UnityClient’",“time”:“Mon, 02 Dec 2019 03:53:43 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘Sending ClientService request for UnityClient’",“time”:“Mon, 02 Dec 2019 03:53:43 GMT”,“level”:“INFO”,“name”:“clientService”}
{“message”:"‘Attempting to retrieve UnrealClient’",“time”:“Mon, 02 Dec 2019 03:53:43 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘Sending ClientService request for UnrealClient’",“time”:“Mon, 02 Dec 2019 03:53:43 GMT”,“level”:“INFO”,“name”:“clientService”}
{“message”:"‘Attempting to retrieve LauncherClient’",“time”:“Mon, 02 Dec 2019 03:53:43 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘Sending ClientService request for LauncherClient’",“time”:“Mon, 02 Dec 2019 03:53:43 GMT”,“level”:“INFO”,“name”:“clientService”}
{“message”:"‘Request ended for UnityClient’",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“clientService”}
{“message”:"‘Error retrieving artifact details from client service - received code NotFound’ { status: 5,\n statusMessage: ‘failed to retrieve Artifact from Assembly Content Service’,\n headers: \n BrowserHeaders {\n keyValueMap: \n { ‘access-control-expose-headers’: [Array],\n ‘content-type’: [Array],\n ‘grpc-message’: [Array],\n ‘grpc-status’: [Array],\n vary: [Array],\n connection: [Array],\n ‘transfer-encoding’: [Array] } },\n message: null,\n trailers: \n BrowserHeaders {\n keyValueMap: \n { ‘access-control-expose-headers’: [Array],\n ‘content-type’: [Array],\n ‘grpc-message’: [Array],\n ‘grpc-status’: [Array],\n vary: [Array],\n connection: [Array],\n ‘transfer-encoding’: [Array] } } }",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“ERROR”,“name”:“clientService”}
{“message”:"‘Got response for UnityClient, Error: Client not found’",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘Request ended for UnrealClient’",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“clientService”}
{“message”:"‘Got response for UnrealClient, [object Object]’",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘Request ended for LauncherClient’",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“clientService”}
{“message”:"‘Error retrieving artifact details from client service - received code NotFound’ { status: 5,\n statusMessage: ‘failed to retrieve Artifact from Assembly Content Service’,\n headers: \n BrowserHeaders {\n keyValueMap: \n { ‘access-control-expose-headers’: [Array],\n ‘content-type’: [Array],\n ‘grpc-message’: [Array],\n ‘grpc-status’: [Array],\n vary: [Array],\n connection: [Array],\n ‘transfer-encoding’: [Array] } },\n message: null,\n trailers: \n BrowserHeaders {\n keyValueMap: \n { ‘access-control-expose-headers’: [Array],\n ‘content-type’: [Array],\n ‘grpc-message’: [Array],\n ‘grpc-status’: [Array],\n vary: [Array],\n connection: [Array],\n ‘transfer-encoding’: [Array] } } }",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“ERROR”,“name”:“clientService”}
{“message”:"‘Got response for LauncherClient, Error: Client not found’",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"'fetched client: {“url”:“https://storage.googleapis.com/improbable-assembly-artifact-storage/beta_blue_robert_270/173550db4367505d12d738f776648783?Expires=1575260024&GoogleAccessId=56260027111-427eh55vcs7gne2i6d6qaa60egsj9k6t%40developer.gserviceaccount.com&Signature=CRKclZl1V9yfNpe0XZ23hPxcAmhfTgGAPZr0pu4SqqKdfFMsa%2F%2Fn%2BV1NZBC%2FlqYQBBiO7rTkEN6lOCN6Cfmhc6KqsiE9AKFPHk%2FV5lR756fbJ0nkyy5KDi%2BA8NsfBk6xZjU6o1Q0q3L58XzGY8VfZFQSVyMWhQvQ7zR%2FN%2FZJfEY%3D",“type”:“UnrealClient”,“md5”:“173550db4367505d12d738f776648783”}’",“time”:"Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘getDir: found md5 173550db4367505d12d738f776648783 at C:\\Users\\windowsUser\\AppData\\Local\\SpatialOS\\Launch\\clients\\173550db4367505d12d738f776648783’",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“DEBUG”,“name”:“cache”}
{“message”:"‘launchClient args: {“token”:“eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRwbCI6IjMwODAyMDI0NDY1MTMxMTQ0NjQiLCJ3a3QiOiIiLCJpYXQiOjE1NzUyNTg4MTEsImV4cCI6MTU3NTI2MDYxMX0.hA4bpXNwdtA0_5q0I80HLKocD3_AdRwIosaaTLU5cHRdVACHX_-cPaqmO6AtYUbRkhPFBBE_zt3CAXSOIeMsfSf4LfgGkryFBSSC-s65Saqq7Ywkd0oQfbFKGDvat7BvvaeJsqApyFMzmeSdLFTBZGUI7_0ZmB-DwP8SNXZ9O8APQ4OB0mVLuI8lxKXcA6XS6f5N_aVqiF2W8GoQAy4IH0ChZkadW68tx2RMG8qt2B23tiqYLJhWea4USd3B83UsKHWpSEhERY2H01CfCa134BTqOu0lUGzNFAHUSBg8N4GDymmpptLLU_k2grB2yEHX3KrjMlbRwRA8ddzTJbo3sQ”,“projectName”:“beta_blue_robert_270”,“clientType”:“UnrealClient”,“clientBasePath”:“C:\\\\Users\\\\windowsUser\\\\AppData\\\\Local\\\\SpatialOS\\\\Launch\\\\clients\\\\173550db4367505d12d738f776648783”,“deploymentName”:“xxxxxxprototypedeployment”,“locatorHost”:“locator.improbable.io”,“infraServicesUrl”:“https://api.spatial.improbable.io”,“playerIdentityToken”:“eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRuIjoiUnlhbiBUaG9ybnRvbiIsImlhdCI6IjIwMTktMTItMDJUMDM6NTM6MzEuMDQ5MjQyMzg0WiIsImV4cCI6IjIwMTktMTItMDNUMDM6NTM6MzEuMDQ5MjQyMzg0WiJ9.F5YofziTBUB2ZSEJOhYPF8yzxeVPvohTUD7oJbCjUxt5hjIK-OIP7apjgpf2-pN4bhx1_BgYtpU7o90kMIH5UOmcIpQhdfUx4Bjd62SJ4_vvzl7SBNWYeGHWneZJplZSqD0oMJdzvWohuehvsruimm7JaAqMYtBhjGzeaCbQbm7BJEpymltuk1wvRY_dpg7n37Xx0Txdc68lGqIhDUadnFeDK2SMKyGNJ78xAIYbfLbDBHIPxlEPDTjiM-vocJKQ_IXg4qGC2RY_vQWwoWrfTJC2QHMbkfPcqlJ0bH-kx6auge9PMWBT4Ve5zqnxkeszycyyIWAGjACThaCBj_imFw”}’",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“launch”}
{“message”:"‘launching with the unreal flow’",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“launch”}
{“message”:"‘found executable files for Unreal: C:/Users/windowsUser/AppData/Local/SpatialOS/Launch/clients/173550db4367505d12d738f776648783/_xxxxxxPrototype.exe,C:/Users/windowsUser/AppData/Local/SpatialOS/Launch/clients/173550db4367505d12d738f776648783/xxxxxxPrototype/Binaries/Win64/xxxxxxPrototype.exe’",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“launch”}
{“message”:"‘launchBinary: path:C:/Users/windowsUser/AppData/Local/SpatialOS/Launch/clients/173550db4367505d12d738f776648783/xxxxxxPrototype.exe args: +appName,beta_blue_robert_270,+projectName,beta_blue_robert_270,+deploymentName,xxxxxxprototypedeployment,+loginToken,eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRwbCI6IjMwODAyMDI0NDY1MTMxMTQ0NjQiLCJ3a3QiOiIiLCJpYXQiOjE1NzUyNTg4MTEsImV4cCI6MTU3NTI2MDYxMX0.hA4bpXNwdtA0_5q0I80HLKocD3_AdRwIosaaTLU5cHRdVACHX-cPaqmO6AtYUbRkhPFBBE_zt3CAXSOIeMsfSf4LfgGkryFBSSC-s65Saqq7Ywkd0oQfbFKGDvat7BvvaeJsqApyFMzmeSdLFTBZGUI7_0ZmB-DwP8SNXZ9O8APQ4OB0mVLuI8lxKXcA6XS6f5N_aVqiF2W8GoQAy4IH0ChZkadW68tx2RMG8qt2B23tiqYLJhWea4USd3B83UsKHWpSEhERY2H01CfCa134BTqOu0lUGzNFAHUSBg8N4GDymmpptLLU_k2grB2yEHX3KrjMlbRwRA8ddzTJbo3sQ,+locatorHost,locator.improbable.io,+infraServicesUrl,https://api.spatial.improbable.io,+playerIdentityToken,eyJhbGciOiJSUzM4NCIsImtpZCI6IjAwMDEifQ.eyJwcnZkciI6ImltcHJvYmFibGUiLCJzdWIiOiIyMDI5NDI5MjY3OTgiLCJwcmoiOiJiZXRhX2JsdWVfcm9iZXJ0XzI3MCIsImRuIjoiUnlhbiBUaG9ybnRvbiIsImlhdCI6IjIwMTktMTItMDJUMDM6NTM6MzEuMDQ5MjQyMzg0WiIsImV4cCI6IjIwMTktMTItMDNUMDM6NTM6MzEuMDQ5MjQyMzg0WiJ9.F5YofziTBUB2ZSEJOhYPF8yzxeVPvohTUD7oJbCjUxt5hjIK-OIP7apjgpf2-pN4bhx1_BgYtpU7o90kMIH5UOmcIpQhdfUx4Bjd62SJ4_vvzl7SBNWYeGHWneZJplZSqD0oMJdzvWohuehvsruimm7JaAqMYtBhjGzeaCbQbm7BJEpymltuk1wvRY_dpg7n37Xx0Txdc68lGqIhDUadnFeDK2SMKyGNJ78xAIYbfLbDBHIPxlEPDTjiM-vocJKQ_IXg4qGC2RY_vQWwoWrfTJC2QHMbkfPcqlJ0bH-kx6auge9PMWBT4Ve5zqnxkeszycyyIWAGjACThaCBj_imFw,+useExternalIpForBridge,true’",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“launch”}
{“message”:"'Attempting to spawn client process. Attempts left: ’ 6",“time”:“Mon, 02 Dec 2019 03:53:44 GMT”,“level”:“INFO”,“name”:“launch”}
{“message”:"‘Close after spawn delay triggered. Calling launchOnce() again’",“time”:“Mon, 02 Dec 2019 03:53:46 GMT”,“level”:“INFO”,“name”:“launch”}
{“message”:"‘launched client’",“time”:“Mon, 02 Dec 2019 03:53:46 GMT”,“level”:“INFO”,“name”:“launcher”}
{“message”:"‘will quit’",“time”:“Mon, 02 Dec 2019 03:53:46 GMT”,“level”:“INFO”,“name”:“launcher”}

I am using the latest Linux Cross_Compile toolchain; 7.0.1
I am using a variation of the one_worker_test.json file where the dimensions have been changed to:
“dimensions”: {
“x_meters”: 10000,
“z_meters”: 10000

I am also simulating 10 extra clients on this map - simulated players.

Hi @ryan_virtuos,

Those USER LOGGING RATE LIMITED - DROPPING LOGS error messages are indicating that your workers are sending too many messages and we’ve had to throttle them to prevent performance issues.
From looking at your cloud deployment it appears that your UnrealWorker has sent nearly 3 million warnings, despite the log rate limiting.

I would think that the next steps to investigate this would be:

  • Fix up as many of those warnings as possible to reduce the visual noise
  • Simplify this scenario to check if you experience a fatal crash without the simulated players

Then it would be useful to take a look at the Unreal client crash dump and logs, as this may point more clearly to the root cause.

Hi all, so I think I have found the root of the problem. Not sure where to go from here. It looks like it’s the following problem from the SpatialOS Launcher’s unreal client:

[2019.12.03-04.05.29:405][111]LogTemp: FWindowsReadRequest request was not aligned. This is expected with loose files, but not a pak file.
[2019.12.03-04.05.29:406][111]LogTemp: FWindowsReadRequest request was not aligned. This is expected with loose files, but not a pak file.
[2019.12.03-04.05.42:652][732]LogWindows: Warning: CreateProc failed: The system cannot find the file specified. (0x00000002)
[2019.12.03-04.05.42:652][732]LogWindows: Warning: URL: …/…/…/Engine/Binaries/Win64/CrashReportClient.exe “C:/Users/windowsUser/AppData/Local/SpatialOS/Launch/clients/0b3ab5351a7a121a5015c21cdcfe141e/BeaconPrototype/Saved/Crashes/UE4CC-Windows-1CE4851F4EB46A95C4A61E9596866798_0000” -AppName=UE4-BeaconPrototype -CrashGUID=UE4CC-Windows-1CE4851F4EB46A95C4A61E9596866798_0000 -DebugSymbols=…\Engine\Intermediate\Symbols
[2019.12.03-04.05.42:652][732]LogWindows: Could not start CrashReportClient.exe
[2019.12.03-04.05.42:652][732]LogMemory: Platform Memory Stats for WindowsNoEditor
[2019.12.03-04.05.42:652][732]LogMemory: Process Physical Memory: 1001.38 MB used, 1725.64 MB peak
[2019.12.03-04.05.42:652][732]LogMemory: Process Virtual Memory: 2556.55 MB used, 2894.16 MB peak
[2019.12.03-04.05.42:652][732]LogMemory: Physical Memory: 18434.80 MB used, 14253.64 MB free, 32688.44 MB total
[2019.12.03-04.05.42:652][732]LogMemory: Virtual Memory: 7392.52 MB used, 134210336.00 MB free, 134217728.00 MB total
[2019.12.03-04.05.50:768][732]LogWindows: Windows GetLastError: The operation completed successfully. (0)
[2019.12.03-04.05.50:768][732]LogThreadingWindows: Error: Runnable thread RenderThread 3 crashed.
[2019.12.03-04.05.50:768][732]LogWindows: Error: === Critical error: ===
[2019.12.03-04.05.50:768][732]LogWindows: Error:
[2019.12.03-04.05.50:768][732]LogWindows: Error: Assertion failed: (GIsCriticalError || !NumMarks) [File:F:\Projects\SpacialOS\Engine\Source\Runtime\Core\Public\Misc/MemStack.h] [Line: 89]
[2019.12.03-04.05.50:768][732]LogWindows: Error:
[2019.12.03-04.05.50:768][732]LogWindows: Error:
[2019.12.03-04.05.50:768][732]LogWindows: Error:
[2019.12.03-04.05.50:768][732]LogWindows: Error:
[2019.12.03-04.05.50:768][732]LogWindows: Error: Crash in runnable thread RenderThread 3
[2019.12.03-04.05.50:777][732]LogOutputDevice: Warning:

Script Stack (0 frames):

[2019.12.03-04.05.50:777][732]LogWindows: Windows GetLastError: The operation completed successfully. (0)
[2019.12.03-04.05.50:777][732]LogWindows: Error: HandleError re-entered.

I’ve cleaned up the errors/warnings on my deployment and am now just down to one warning:
“Trying to set unknown deployment flag ‘streaming_query_interval’ to ‘4’. If you configured this flag in the legacy flags section of your launch configuration, make sure you spelled its name correctly.”
Which I assume is due to the switch over to the flexible project layout, where the flag seems to be different:


Getting back to the main issue though, I still crash out and it is only when I launch from the console and kick off the launcher. I don’t get this issue through a local deployment kicked off by the editor - either in PIE or when Launching the exe locally from the project.

It seems like it’s looking for a pak file, but the project the launcher pulls down is not in pak format. I have gone over the deployment instructions again just in case I was missing something, but still have the problem.

location2

Instructions followed:

I’ve tried packaging using your typical UE4 method for packaging for windows as a test to try to narrow things down, and I find that I don’t have these problems.

If I run the client directly from my AppData folder, it’ll crash out as before. I feel like there’s something particular about these spatialos build scripts that’s causing this problem. Please advise.

Hi @ryan_virtuos

Would you mind if I spooled up your deployment and attempted to connect to it myself to see if I experience the same crash?

Thanks

Please do!