Java no space left on device error

Jira server functionality fails with No space left on device error

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the server and data center platforms .

Problem

All sorts of functionality in JIRA may fail once the disk runs on out space. The type of functionality that fails depends upon how the disks have been configured — for example:

• JIRA may fail to startup
• Items in the UI may be missing
• Attachments cannot be uploaded
• Reindexing may fail or perform very slowly

Depending on which partition is out of space (the JIRA-Home or JIRA-Install), JIRA will not be able to start up or items in the UI will be missing.

The following appears in the atlassian-jira.log :

Caused by: java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:282) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:1

Diagnosis

If this exception is thrown, it indicates there’s a problem with disk space. Locate the JIRA-Home and JIRA-Install directories and verify the amount of space left on these partitions. Additionally ensure the database server has sufficient disk space available.

Cause

One of the two partitions, or the database server, is out of space, or does not have enough left for JIRA to operate normally.

Resolution

JIRA will not necessarily gracefully recover from a disk space outage, it is highly likely corruption will occur. Databases are hardened against this, however JIRA is not designed to be able to be run with insufficient space on the disk.

1. Immediately stop JIRA.
2. Hosting JIRA indexes ( $JIRA_HOME/caches/indexes ) over network shares is not officially supported as per our Supported platforms . If using a remote network share please move this to a locally mounted drive.
3. Allocate more space where the directories and/or database is located and restart JIRA.
4. Restore the last known working backup from prior to the outage.
5. Install monitoring for the disk space to prevent this problem from happening again.

Источник

Indexer stops with ERROR java.io.IOException: No space left on device

Platform notice: Server and Data Center only. This article only applies to Atlassian products on the server and data center platforms .

Summary

Newly added content is unsearchable because Indexer stopped working after system ran out of memory.

Bellow are some of the exceptions that can be found in atlassian-confluence.log

ERROR [Indexer: 1] [internal.index.lucene.LuceneBatchIndexer] doIndex Exception indexing document with id: 12345678 │java.io.IOException: No space left on device at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAccessFile.write(RandomAccessFile.java:525) at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:474) at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:113)

ERROR [scheduler_Worker-10] [org.quartz.core.ErrorLogger] schedulerError Job (DEFAULT.IndexQueueFlusher threw an exception. org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: com.atlassian.bonnie.LuceneException: com.atlassian.confluence.api.service.exceptions.ServiceException: Failed to process entries] . Caused by: com.atlassian.confluence.api.service.exceptions.ServiceException: Failed to process entries . Caused by: java.io.IOException: No space left on device

WARN [scheduler_Worker-10] [search.lucene.tasks.AddDocumentIndexTask] perform Error getting document from searchable java.io.IOException: No space left on device WARN [scheduler_Worker-1] [search.lucene.tasks.AddDocumentIndexTask] perform Error getting document from searchable java.io.FileNotFoundException: /opt/confluence/confluence-home/index/_z8ey.fdx (No space left on device) at java.io.RandomAccessFile.open0(Native Method) at java.io.RandomAccessFile.open(RandomAccessFile.java:316) at java.io.RandomAccessFile.(RandomAccessFile.java:243) at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:466)

Diagnosis

On Linux

• Increase disk capacity or analyze which directories / files are large and can be removed
• If you ran out of inodes, you might need to migrate to another file system or partition, see ‘Not enough space on disk’ errors in Confluence 6.5 or later when disk space is available

On Windows

• Check free disk space from properties of the drive
• Check free disk space from PowerShell

Get-WmiObject -Class Win32_logicaldisk

Cause

As documented in ‘java.io.IOException No space left on device’ due to disk space KB, «No space left on device» exception indicates that there is not enough space left for Confluence to write inside directory.

Besides failures to generate backups, or save attachments, indexing tasks will also fail .

Solution

Confluence stores it’s index locally, inside home directory, and it needs to have enough free space allocated to continue indexing content.

After enough space is freed, rebuild index from scratch by following one of the bellow procedures:

Lucene index must be stored locally on the server and can’t be stored on NFS mount as outlined in Health Check: Lucene index files location

Источник

Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

java.io.IOException: No space left on device #812

java.io.IOException: No space left on device #812

Comments

I am getting the dreaded no space left on device error despite having set up local.dir in my config file

where mypath has more than 5TB of free space.

I am running something really simple like

 df_new <-spark_read_parquet(sc, "/mypath/parquet_*", name = "df_new", memory = FALSE) myquery <- df_new %>% group_by(text) %>% summarize(mycount = n()) %>% arrange(desc(mycount)) %>% head(10) 

running collect(myquery) returns the error. Is there a sparklyr parameter that overrides the writing of temporary files to the /mypath folder? Any ideas @kevinykuo @javierluraschi ?

The text was updated successfully, but these errors were encountered:

@kevinushey @javierluraschi could it be that sparklyr does not allow for different group_by methods such as groupByKey and ReduceByKey (this one being optimized for skewed data like mine) ?

Although I’m not able to give you a direct answer, I would start by inspecting the generated SQL:

myquery % group_by(text) %>% summarize(mycount = n()) %>% arrange(desc(mycount)) %>% head(10) sql_render(myquery) 

Does the generated SQL look correct?

@kevinushey I actually get some crazy stuff

> myquery % group_by(u) %>% summarize(mycount = n()) %>% + arrange(desc(mycount)) %>% head(10) > sql_render(myquery) SELECT * FROM (SELECT * FROM (SELECT `u`, count(*) AS `mycount` FROM `df_new` GROUP BY `u`) `jpdowetklw` ORDER BY `mycount` DESC) `jhdbjylver` LIMIT 10 

u is a url address in my data. what is this jhdbjylver crazyness?? 😀 Any ideas?

@randomgambit jpdowetklw is a named subquery that dplyr uses to disambiguate names. Could you share the full stack trace of the No space left on device exception?

 Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 5231 in stage 4.0 failed 4 times, most recent failure: Lost task 5231.3 in stage 4.0 (TID xxxx, xx.xx.xxx.xx, executor xx): java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:345) at org.apache.spark.storage.TimeTrackingOutputStream.write(TimeTrackingOutputStream.java:58) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at net.jpountz.lz4.LZ4BlockOutputStream.flush(LZ4BlockOutputStream.java:225) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$2.flush(UnsafeRowSerializer.scala:91) at org.apache.spark.storage.DiskBlockObjectWriter.commitAndGet(DiskBlockObjectWriter.scala:170) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:157) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 

@randomgambit does this happen to run correctly?

DBI::dbGetQuery(sc, "SELECT `u`, count(*) AS `mycount` FROM `df_new` GROUP BY `u` ORDER BY `mycount` DESC")

This would be the simplified query, wondering if the nested select dplyr is generating is causing a performance degradation here.

@javierluraschi let me try this. but arent you missing a LIMIT somewhere? my query has a head() at the end. thanks!

Источник

Решение

Нехватка места на диске — одна из наиболее распространенных причин проблем в любой производственной системе, и Jenkins — не исключение. Если вы столкнулись с ошибками «на устройстве не осталось места», процесс устранения проблемы относительно прост.

1. В сообщении об ошибке должно быть указано, куда система пыталась записать; Например, /var/jenkins_home
2. В уязвимой системе (либо мастер Jenkins, либо, возможно, агент) запустите df и найдите тома, % использования которых равен 100%.
3. Для томов, которые используют 100 % блочного хранилища или приближаются к нему, вы можете искать «большие» файлы, используя файлы find . Например, find /var/jenkins_home -size +100M . Или вы можете использовать du -sh * в каталоге и соответственно развернуть. Оба они могут быть очень медленными, если объем большой.
4. Если обычный df том не показывает никаких томов при 100% использовании, возможно, в рассматриваемом томе нет инодов. df -i покажет это как IUse% на 100%.
5. Когда в томе нет инодов, но кажется, что он имеет достаточно свободного блочного хранилища, это обычно вызвано наличием очень большого количества маленьких или пустых файлов или каталогов. Каждый созданный файл или каталог потребляет 1 inode, и они являются конечным ресурсом. Чтобы найти, где это может происходить, вы можете зайти в каждый каталог тома и выполнить простой find . | wc -l поиск необычно больших чисел. Опять же, это может потребовать углубления, чтобы найти источник проблемы.
6. Наконец, если ни одна из этих проверок ничего не дает, возможно, задание Jenkins или другой процесс пытается создать один файл, объем которого превышает объем свободного места. Когда том заполнен, процесс завершается с ошибкой, и файл удаляется, поэтому кажется, что проблемы нет. В этом случае для обнаружения вызывающего нарушение задания или процесса потребуется просмотр журналов Jenkins или системы.

Источник